Top

'top' - display Linux processes.

It is part of the procps package, a set of Linux utilities that provide system information. And other tools such as free, vmstat, ps, and more. Top gets most of its info from several files under the /proc directory. It pretty much summarizes most of them into one window.

The Basics

CPU

In accordance with the USE method, I'm going to split this up based on the different resources, starting with CPU.

We'll be going more in depth into CPU architecture later, but the important basics to know here are that, as far as performance goes, there are three basic buckets we worry about on a CPU.

Computation: This is the part of the CPU doing the actual math. In broad terms, a CPU performs a certain amount of computation each cycle, and the clock rate is how quickly it can perform cycles. A 4GHz processor can perform 4,000,000,000 cycles per second, 800,000,000 more than a 3.2GHz processor. The latency on cycles is fixed, so throughput here is a measure of how many the processor is capable of doing. To note, not all cycles are equal. Processors of different generations and models can all perform a different amount of instructions per cycle (IPC), which influences the total amount of work being done.

Memory: While CPUs do not have main memory, they do have registers and multiple levels of cache. The primary difference between the memory on a CPU and the memory off of it is latency. Each time you go up a layer in cache, latency increases. When you go to main memory, the RAM in the system, latency significantly increases. In modern CPUs, things such as the L1, L2 caches and translation lookaside buffers are per-core. The larger L3 cache is shared across the entire CPU by default. We will not be covering how to measure CPU memory utilization in this, but it is important to remember.

Bus: The processor must communicate with other devices on the system, and is limited in the speed and maximum throughput. This includes communication to things like main memory, PCIe devices, etc. Like CPU memory, we will not be covering how to measure this, but it is important to remember.

Hyperthreading is the Intel version of Simultaneous Multi-Threading (SMT). It allows for the execution resources of a physical core (the execution engine - part that does the math) to be shared between two copies of the hardware that stores the architectural state. This allows for two threads to be utilized per physical core. Generally, there are performance benefits to this, but due to the possibility of the execution resources being over-scheduled between the hyperthreads, some use cases will see performance improve when it is disabled. Modern OS schedulers will generally attempt to prevent "hot" threads from being scheduled onto hyperthread siblings. For further details, Anandtech's article on Hyper-Threading is quite good, even with it's age.
- Something to note with special regards to EC2: A vCPU is a hyperthread of a core. That means that an 8 vCPU instance has 4 physical CPU cores backing it. Keep this in mind!

Threads - a single line of commands that are getting processed (or an instruction in other words).

Top from the Top-Down

The first line shows: The time (hour:min:sec), the uptime (hour:min), the total number of active users, and the system load average.

The second line gives a breakdown of the number of tasks, next is the CPU stats, and then the current memory usage.

top - 00:12:45 up 13 days, 10:06,  1 user,  load average: 0.26, 0.26, 0.16
Tasks: 123 total,   1 running, 122 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.2 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7706016 total,   972052 free,   663460 used,  6070504 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  6371624 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0  125596   4024   2604 S   0.3  0.1   4:04.50 systemd
 1918 clouden+  23   3 2593660 218476  11312 S   0.3  2.8   8:49.60 java
 7693 ardika    20   0  162000   2232   1568 R   0.3  0.0   0:00.86 top
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.12 kthreadd
    4 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    6 root      20   0       0      0      0 S   0.0  0.0   0:08.38 ksoftirqd/0
    7 root      rt   0       0      0      0 S   0.0  0.0   0:01.96 migration/0
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      20   0       0      0      0 S   0.0  0.0   0:34.70 rcu_sched
   10 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drai

System Load Average

top - 00:12:45 up 13 days, 10:06,  1 user,  load average: 0.26, 0.26, 0.16

Load Average is the representation of the number of processes running (using a slice of CPU time) AND waiting (queued) to be run. The load average includes processes running the CPU, waiting on disk IO, or network IO (confirm this?). They are the three numbers averages over one, five, and fifteen minute averages. These numbers provide a quick glance at overall system load - they are NOT CPU specific.

Other definitions:

Load average is displayed for 1 minute, 5 minute, and 15 minute intervals. Roughly. This isn't actually entirely true, but it's close enough for getting a rough idea of what's going on with the system, which is about the extent of load average as a useful metric. You should not ever think about it as an exact metric that tells you the full story of what is happening in a system.
In Linux, load averages are “system load averages,” for the system as a whole, measuring the number of threads that are working AND waiting to work (CPU, disk, uninterruptible locks). It measures the number of threads that aren’t completely idle.
System Load Average = is equal to the average number of processes in a runnable or uninterruptible state. Runnable processes are either currently using the CPU or waiting to do so, and uninterruptible processes are waiting for I/O.

Two things contribute to the load average metric:

CPU tasks in a a 'runnable' state - tasks that want to run on a CPU
Tasks in an 'uninterruptable' state - tasks that cannot be interrupted off of the CPU.

Generally due to pending IO, but not always - modern Linux kernels include additional ways that a process might be accounted for as uninterruptable, but these are more rare than processes waiting on IO. (A thread that can be interrupted is a thread that can be moved off of being processed by a CPU to allow another thread to execute)

This makes determining what an "acceptable" load average is a little difficult. The general rule of thumb is that a system being fully utilized will have a load average that is about 1:1 with the number of cores in the system. In reality, this isn't quite the case as the math that makes these numbers appear will result in a bit of smearing due to the moving averages, how they're sampled, and how they're calculated. It also doesn't take into account tasks that are uninterruptable, so you could see a load average that is significantly lower than the number of cores, but still be seeing performance problems.

(In Unix this is referred to the run-queue length: the sum of the number of processes that are currently running plus the number that are waiting (queued) to run.)

In a “single-core CPU” system:
- 0.00 means there is no traffic, or no processes that are currently running. (Idle)
- 1.00 means the CPU load is running exactly at capacity, or is under constant load.
- Over 1.00 means there is back up, or processes queued up/waiting to run.

Multi-processors:
- On multi-processor system, the load is relative to the number of processor cores available.
- 100% utilization mark is 1.00 on single-core system, 2.00 on dual-core, 4.00 on a quad-core, etc.
- Same goes with CPUs: load of 1.00 is 100% of CPU utilization on box with one CPU, 2.00 is 100% CPU utilization on a box with 2 CPUs).
The total number of cores is what matters, regardless of how many physical processors those cores are spread across.

“Number of cores = max load” Rule of Thumb: on a multicore system, your load should not exceed the number of cores available.

How to find the number of cores:

In 'top', you can press '1' to see the number of CPU(s)/core(s).
nproc - print the number of processing units available
cat /proc/cpuinfo to get info (model name, CPU cores, cache size, etc.) on each processor in your system. Just to get a count of CPUs:

nproc
grep 'model name' /proc/cpuinfo | wc -l

In a system with 1 CPU core, is 1.00 the ideal load? No, there is no headroom. Many sysadmins will draw the line at 0.70.

“Need to look into” Rule of Thumb: 0.70. If your load average is staying above > 0.70, it’s time to investigate before things get worse.
“Fix this now” Rule of Thumb: 1.00. If your load average stays > 1.00, find the problem and fix it now!
“Arrgh, it’s 3AM WTF?” Rule of Thumb: 5.0. If your load average is > 5.00, you could be in serious trouble. Your box is either hanging or slowing way down. Don’t let it get to this point.

Which average should you be focusing on? One, five, or fifteen minute?

You’d want to focus on the 15 min averages. If your box spikes above 1.0 on the one minute average its fine. Its when the 15 min average goes above 1.0 is when you want to worry. (Obvi, adjust the load max = number of processor cores your system has)

If averages are 0.0, then your system is idle.
If the 1 minute average is higher than 5 or 15 minute averages, then load is increasing.
If the 1 minute is lower than 5 or 15 minute, the load is decreasing.
If averages are higher than your CPU/core count, then you may have a performance problem.

/proc/loadavg - This file provides a load average in regard to both the CPU and IO over time, as well as additional data used by uptime and top. A sample /proc/loadavg looks like:

0.20 0.18 0.12 1/80 11206

The first three columns measure CPU and IO utilization of the last one, five, and 15 minute periods. The fourth column shows the number of currently running processes and the total number of processes. The last column displays the last process ID used.

What is a high load average? In other words, what load average do you consider to be high? It depends on what is causing it. A spike in load could mean a few things, so it's important do determine whether the load is either:

CPU-bound (processes waiting on CPU resources)
RAM-bound (specifically high RAM usage that has moved into swap(swap space))
I/O-bound (processes fighting for disk or network I/O)

The Tasks Line

(The 4(+2) Process states)

Tasks: 123 total,   1 running, 122 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.2 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

The task line shows statistics about number of tasks in various states. Shows total tasks or threads, depending on the state of the Threads-mode toggle. That total is further classified as:

running (R): Represents the number of tasks in a runnable state. (Doesn't necessarily mean its currently executed, more like processes ready to run/be executed. Just waiting for the scheduler to decide when to run.) Runnable state is when process has got all the system resources to perform its operation except CPU. The process is ready to go once the CPU is free.

sleeping: Processes being blocked due to waiting for an event (i.e. timeout, or I/O completion). Sleeping process is the one who awaits for resources to run. Since its waiting, it gives up CPU and goes to sleep mode. Once its required resource is free, it gets placed in the scheduler queue for CPU to execute. They don't consume the CPU time, but they do take up memory. There are two types of sleep modes:
- Interruptible sleep mode (S) - In this mode process awaits for particular time slot or specific event to occur. If those conditions occur, then the process will come out of sleep mode.
- Uninterruptible sleep mode (D) - An Uninterruptible sleep state is one that won't handle a signal right away. It will wake only as a result of a waited-upon resource becoming available or after a time-out occurs during that wait (if the time-out is specified when the process is put to sleep). These are processes that cannot be killed or interrupted with a signal. For example if process waiting for disk I/O or any other I/O, process is put in in D state.

stopped: A stopped process is one that has been put to sleep manually. The exact meaning here is "paused", not "terminated." A process that is "stopped" is a process that is suspended by receiving a SIGSTOP signal (or pressing Ctrl+Z or $ kill -STOP ${PID}). This tells the kernel to not perform any processing on it as it has been stopped, and it can only be resumed by sending a SIGCONT signal. Basically stopped process awaits a continuation signal from the kernel, similarly as suspended process awaits a wake-up condition from the kernel.

zombie: (Z) a process whose execution is completed but it still has an entry in the process table. Zombie processes usually occur for child processes, as the parent process still needs to read its child’s exit status. Once this is done using the wait system call, the zombie process is eliminated from the process table. The act of reading that exit code is called "reaping" the child. Between the time a child exits and is reaped, it is called a zombie.
- Are Zombie processes bad or good? Do they impact system performance? Zombie processes don't use any system resources but they do retain their process ID in the process table. If there are a lot of zombie processes, then all the available process ID’s are monopolized by them. This prevents other processes from running as there are no process ID’s available.

The CPU Line

(CPU Stats)

%Cpu(s):  0.2 us,  0.2 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

CPU percentage is the amount of a TIME that the system's processes were found to be active on the CPU. CPU percentage is a function of time. It gives you information about what the CPUs are currently doing, essentially.

"us" - The amount of the CPU time running user space processes that are NOT "niced". This is any process that doesn't belong to the kernel such as Shells, compilers, databases, web servers, and the programs associated with the desktop are all user space programs.

"sy" - The SYstem CPU time. The amount of time that the CPU spent running the kernel and kernel processes.
- All the processes and system resources are handled by the Linux kernel. When a user space process needs something from the system such as allocation of memory, perform I/O, or it needs to create a child process, then the kernel is running.
- In fact the scheduler itself which determines which process runs next is part of the kernel.
- The amount of time spent in the kernel should be as low as possible. This number can peak higher especially if there are a lot of I/O happening.

"ni" - Niceness. The priority level a user space process has, which can be tweaked. The ni stat shows how much time the CPU spent running user space process that have been "niced." Nicing is a process that allows you to change its priority in relation to other processes.

"id" - The amount of time the processor was idle. You want this to be high.
- If you have a sluggish system and this number is high, then you know the cause isn't high CPU load.

"wa" - I/O wait. This is the amount of time the CPU is spent waiting for an I/O operation to complete.
- Input and output operations, like reading or writing to a disk, are slow compared to the speed of a CPU. There are times when the processor has initiated a read or write operation and then it has to wait for it to complete so it has nothing else to do but idle.
- This is valuable when you want to track down the cause of a sluggish system. If this value is low, you can safely rule out disk or network I/O as the cause.

"hi" - Hardware Interrupts; Percentage of CPU time spent servicing hardware interrupts.
- Hardware interrupts are physical interrupts signals sent to the CPU from peripherals like disks and network interfaces.
- Hardware interrupt will actually cause the CPU to stop what it is doing to go handle the interrupt.
- This could be an indication of a broken peripheral that is causing lots of hardware interrupts (hi).

"si" - Software Interrupts; Percentage of CPU time spent servicing software interrupts.
- Software interrupts come from processes and programs running on the system.
- Software interrupts occur at the kernel level not CPU level, so it doesn't stop the CPU.

"st" - The amount of CPU time that was stolen from your virtual machine by the hypervisor, in order for other tasks, such as servicing other VMs or for its own needs.

CPU Load Averages vs CPU Percentage (2 big difference):

Load averages measure the trend in CPU utilization not only an instantaneous snapshot, as does percentage.
Load averages include all demand for the CPU not only how much was active at the time of measurements.

The Total percentage of “us” + “ni” + “id” (and sy?) = should be close to 100%.

The Memory Usage Section

KiB Mem :   984180 total,   237860 free,   500476 used,   245844 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   343596 avail Mem

This section displays the amount of free and used memory in the system. (See the free command page)

Per-Process Statistics

The bottom half of top:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM      TIME+  COMMAND
9463 mysql     16   0  686m 111m 3328 S   53  5.5  569:17.64  mysqld
18749 nagios     1   0  140m 134m 1868 S   12  6.6    1345:01  nagios2db_status
24636 nagios    17   0 34660  10m  712 S    8  0.5    1195:15  nagios
22442 nagios    24   0  6048 2024 1452 S    8  0.1    0:00.04  check_time.pl

In this example, the mysqld process is consuming 53% of the CPU and the nagios2db_status process is consuming 12%. Note that this is the percentage of a single CPU, so if you have a four-CPU machine, you could possibly see more than one process consuming 99% CPU.

PID - Process ID.
USER - Th username of whoever launched the process.
PR - Priority : The priority of the process. Processes with higher priority will be favored by the kernel and given more CPU time than processes with lower priority. The highest priority on *nix is -20 and the lowest is 20.
NI - Nice value : nice is a way of setting your process' priority. Niceness range is from -20 (highest priority) to 19.

(NI is the nice value, which is a user-space concept. PR is the process's actual priority, as viewed by the Linux kernel.)

VIRT - Virtual Memory Size (KiB) : The total amount of virtual memory used by the process
- stands for the virtual size of a process, which is the sum of memory it is actually using, memory it has mapped into itself (for instance the video card's RAM for the X server), files on disk that have been mapped into it (most notably shared libraries), and memory shared with other processes.
RES - Resident Memory Size (KiB) : The non-swapped physical memory a task has used.
SHR - Shared Memory Size (KiB) : The amount of shared memory available to a task, not all of which is typically resident. It simply reflects memory that could be potentially shared with other process.
S - Process Status : The status of the task which can be one of:
- 'D' = uninterruptible sleep
- 'R' = running
- 'S' = sleeping
- 'T' = traced or stopped
- 'Z' = zombie
%CPU - CPU Usage : The percentage of your CPU that is being used by the process. By default 'top' displays this as a percentage of a single CPU.
- On multi-core systems, you can have percentages that are greater than 100%. For example,. if 3 cores are at 60% use, 'top' will show a CPU use of 180% (for details)
%MEM - Memory Usage (RES) : A task's currently used share of available physical memory (RAM).
TIME+ - CPU Time, hundredths : Total CPU time the task has used since it started.
COMMAND - Command Name or Command Line : To see the full command line that launched the process, start top with the -c flag :

$ top -c

top - 04:34:24 up 6 days, 22:57,  1 user,  load average: 0.10, 0.03, 0.01
Tasks: 110 total,   1 running,  70 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.2 sy,  0.0 ni, 99.5 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :   984180 total,   218944 free,   477688 used,   287548 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   367600 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2264 root      20   0  159252  29532   6320 S   0.3  3.0  30:51.16 /opt/aws/amazon-cloudwatch- agent/bin/amazon-cloudwatch-agent -config /opt/aws/amazon-cloudwatch-agent/etc/amazo+
14385 mysql     20   0 1299640 102636  14384 S   0.3 10.4   0:40.83 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/v+
    1 root      20   0  191188   3676   2096 S   0.0  0.4   1:27.88 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.09 [kthreadd]
    4 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 [kworker/0:0H]
    6 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 [mm_percpu_wq]
    7 root      20   0       0      0      0 S   0.0  0.0   0:00.25 [ksoftirqd/0]
    8 root      20   0       0      0      0 I   0.0  0.0   0:15.23 [rcu_sched]
    9 root      20   0       0      0      0 I   0.0  0.0   0:00.00 [rcu_bh]
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.05 [migration/0]
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.82 [watchdog/0]
   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 [cpuhp/0]

To sort process by RAM or memory usage, hit "shift + "m".

If you find a process consuming up all the RAM, you can either kill the process, or you might need to do more troubleshooting to determine what is causing that process to use so much RAM.
Click here for more information about out of memory troubleshooting.

How to use Top to check individual process memory usage (VIRT)

Top provides a dynamic real-time view of the running system, and it gives the CPU and memory usage for the whole system, along with for every program, in percentage:

To filter by a program PID:

top -p <PID>

Example:

Find the process ID (PID) of a process:

$ ps aux | grep falcon
root     1941908  0.0  0.0   8740   108 ?        Ss   Feb29   0:00 /opt/CrowdStrike/falcond
root     1941909  0.0  0.8 1272836 32664 ?       Sl   Feb29   1:19 falcon-sensor
asuslis+ 2088620  0.0  0.0  12144  1096 pts/0    S+   19:29   0:00 grep --color=auto falcon

Get the VIRT of the process:

top -p 1941909

Output shows the falcon-sensor process is using about 8740 KiB (8.74 MiB) total amount of virtual memory:

top - 19:30:11 up 39 days,  4:35,  1 user,  load average: 0.00, 0.00, 0.00
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.2 hi,  0.0 si,  0.0 st
MiB Mem :   3650.6 total,   1108.6 free,    604.2 used,   1937.7 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   2737.1 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1941908 root      20   0    8740    108      0 S   0.0   0.0   0:00.00 falcond

References: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Reference_Guide/s2-proc-loadavg.html https://learning.oreilly.com/library/view/devops-troubleshooting-linux/9780133035513/ch02.html https://kerneltalks.com/linux/process-states-in-linux/ http://fibrevillage.com/sysadmin/462-how-to-identify-processes-that-waiting-for-disk-i-o https://www.tutorialspoint.com/what-is-zombie-process-in-linux https://stackoverflow.com/questions/20688982/zombie-process-vs-orphan-process http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html https://superuser.com/questions/575202/understanding-top-command-in-unix/575330