CPU/RAM Troubleshooting

How to display the amount of logical processors

[root@node01 ~] psrinfo -vp
The physical processor has 4 cores and 8 virtual processors (1 3 5 7 9 11 13 15)
    The core has 2 virtual processors (1 9)
    The core has 2 virtual processors (3 11)
    The core has 2 virtual processors (5 13)
    The core has 2 virtual processors (7 15)
x86 (GenuineIntel 106A5 family 6 model 26 step 5 clock 2267 MHz)
  Intel(r) Xeon(r) CPU           E5520  @ 2.27GHz
The physical processor has 4 cores and 8 virtual processors (0 2 4 6 8 10 12 14)
    The core has 2 virtual processors (0 8)
    The core has 2 virtual processors (2 10)
    The core has 2 virtual processors (4 12)
    The core has 2 virtual processors (6 14)
x86 (GenuineIntel 106A5 family 6 model 26 step 5 clock 2267 MHz)
  Intel(r) Xeon(r) CPU           E5520  @ 2.27GHz

How to display CPU statistics and load distribution among processors

[root@node01 ~] mpstat 1 5
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  870   1  274  2440  235  569    2   47   59    0  4709    2   3   0  94
  1  547   1   58   203   63  462    3   19   35    0  4964    2   2   0  95
  2  326   0  142   424  132  690    1   29   52    0  2427    1   2   0  97
  3  168   1   38   325  123  814    1   17   39    0  1875    1   2   0  97
  4  105   0   31   326  135  224    0   16   37    0  1060    1   1   0  98
  5  114   0   25   216   87  382    1   16   31    0  1192    1   1   0  98
  6  116   0   38   459  153  769    1   14   37    0  2732    1   2   0  97
  7  241   1   20   656  185  762    1   15   33    0  1466    0   1   0  98
  8   95   0   34   230   92  327    1    9   26    0  1620    1   1   0  98
  9  111   0   18   238   86  691    1   15   34    0  1481    1   1   0  98
  10   66   0   37   160   64  219    0    7   25    0  1184    0   1   0  99
  11  103   0   21   157   61  345    1   12   26    0  1281    0   1   0  99
  12   50   0   20   721  194  306    1   10   28    0  1318    0   1   0  98
  13  129   0  203   320  141  741    1   15   33    0  1445    1   1   0  98
  14   34   0   13   213   72  342    0    7   25    0  1453    0   1   0  99
  15   94   0   20   406  226  367    1   12   54    0   883    0   1   0  98
  • CPU - Processor ID.

Note

A high number of following calls can be an indication of performance problems.

  • xcal - Number of inter-processor cross-calls.
  • intr - Number of interrupts.
  • smtx - Number of operations, which couldn’t acquire a lock on the first try (spins on mutexes).
  • syscl - Number of syscalls.
  • usr - Percent of processor user time.
  • sys - Percent of processor system time.
  • idl - Percent of processor idle time.

How to display the amount of total memory available in the system (MiB)

[root@node01 ~] prtconf -m
32758

How to display virtual memory usage

[root@node01 ~] vmstat
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr lf rm s0 s1   in   sy   cs us sy id
 0 0 0 54267716 25895544 366 3131 74 0 0 0 240 6 -268 186 -1 7442 30957 7950 1 1 98
  • r - The number of kernel threads in run queue. High values of r may indicate performance issues.
  • us - User time (%).
  • sy - System time (%).
  • id - Idle time (%).

How to display processes on the system and in the zones

[root@node01 ~] prstat -Z
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  3538 root     2095M 2083M sleep    1    0   0:03:54 0.4% qemu-system-x86/12
  3421 root     2094M 2082M sleep   59    0   0:01:09 0.1% qemu-system-x86/5
    90 root        0K    0K sleep   99  -20   0:00:18 0.0% zpool-zones/238
  3601 root       47M   31M sleep   59    0   0:00:07 0.0% python2.7/1
  3585 root       38M   28M sleep   59    0   0:00:03 0.0% python2.7/1
  3469 root       37M   28M sleep   59    0   0:00:03 0.0% python2.7/1
  2843 root       14M   11M sleep    1    0   0:00:03 0.0% kvmiostat/1
  3632 root       42M   17M sleep    1    0   0:00:00 0.0% python2.7/1
  9573 root     4560K 3408K cpu1     1    0   0:00:00 0.0% prstat/1
   184 root     6156K 2948K sleep    1    0   0:00:00 0.0% syseventd/18
  2829 root     3996K 2436K sleep    1    0   0:00:00 0.0% vm-kvm-disk-io-/1
  2806 root     6092K 4600K sleep    1    0   0:00:00 0.0% vm-network-moni/1
  2203 root     2604K  812K sleep   29    0   0:00:00 0.0% lldpd/1
  2801 root     1876K 1180K sleep   59    0   0:00:00 0.0% ctrun/1
  2738 root     3952K  892K sleep   59    0   0:00:00 0.0% ipmon/1
  2709 root     4888K 3560K sleep    1    0   0:00:00 0.0% picld/4
  2821 root     2120K 1084K sleep   59    0   0:00:00 0.0% cron/1
  2852 daemon   4116K 1812K sleep    1    0   0:00:00 0.0% rpcbind/3
  3560 root       35M   16M sleep    1    0   0:00:00 0.0% python2.7/1
  2834 root     5144K 1216K sleep   59    0   0:00:00 0.0% zabbix_agentd/1
    88 root     2664K 1524K sleep   29    0   0:00:00 0.0% pfexecd/3
    21 root     2988K 1608K sleep   29    0   0:00:00 0.0% dlmgmtd/7
  2825 root     4868K 3392K sleep    1    0   0:00:00 0.0% vm-cpu-monitor/1
  2835 root     5144K 1216K sleep   59    0   0:00:00 0.0% zabbix_agentd/1
ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE
     3        2 2095M 2083M   6.4%   0:03:54 0.4% f7860689-c435-4964-9f7d-2d2*
     1        2 2094M 2082M   6.4%   0:01:09 0.1% a28faa4d-d0ee-4593-938a-f0d*
     0       89 1330M  629M   1.8%   0:00:50 0.1% global

Total: 93 processes, 521 lwps, load averages: 0.25, 0.26, 0.26
  • PID - Process ID.

  • USERNAME - The real user (login) name or real user ID.

  • SIZE - The total virtual memory size of the process, including all mapped files and devices.

  • RSS - The resident set size of the process (RSS), in kilobytes (K), megabytes (M), or gigabytes (G).

  • STATE - The state of the process.

    • cpuN - Process is running on CPU N.
    • sleep - Sleeping: process is waiting for an event to complete.
    • wait - Waiting: process is waiting for CPU usage to drop to the CPU-caps enforced limit.
    • run - Runnable: process in on run queue.
    • zombie - Zombie state: process terminated and parent not waiting.
    • stop - Process is stopped.
  • TIME - The cumulative execution time for the process.

  • CPU - The percentage of recent CPU time used by the process.

  • PROCESS - The name of the process.

How to display memory usage of virtual machines

[root@node01 ~] zonememstat
                                 ZONE  RSS(MB)  CAP(MB)    NOVER  POUT(MB)
                               global        0        -        -         -
 a28faa4d-d0ee-4593-938a-f0d062022b02     2082     3072        0         0
 f7860689-c435-4964-9f7d-2d2d70cfe389     2082     3072        0         0
  • ZONE - The zone name.
  • RSS - The amount of physical memory consumed.
  • CAP - The memory cap.
  • NOVER - Number of times the zone reached over its cap.
  • POUT - The amount of paged out memory.

How to display memory usage by type

[root@node01 ~] echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     462970              1808    6%
ZFS File Data              127036               496    2%
Anon                      1169355              4567   14%
Exec and libs                4516                17    0%
Page cache                  15842                61    0%
Free (cachelist)            19856                77    0%
Free (freelist)           6584225             25719   79%

Total                     8383800             32749
Physical                  8383799             32749

How to display the amount of consumed memory by processes and virtual machines

[root@node01 ~] prstat -s rss -z 3
PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
3538 root     2095M 2083M sleep    1    0   0:04:11 0.5% qemu-system-x86/12
3274 root        0K    0K sleep   60    -   0:00:00 0.0% zsched/1

Total: 2 processes, 13 lwps, load averages: 0.26, 0.26, 0.26

Interrupt mapping

[root@node01 ~] echo ::interrupts | mdb -k
IRQ  Vect IPL Bus    Trg Type   CPU Share APIC/INT# ISR(s)
3    0xb1 12  ISA    Edg Fixed  10  1     0x0/0x3   asyintr
4    0xb0 12  ISA    Edg Fixed  9   1     0x0/0x4   asyintr
9    0x81 9   PCI    Lvl Fixed  1   1     0x0/0x9   acpi_wrapper_isr
17   0x85 9   PCI    Lvl Fixed  6   1     0x0/0x11  uhci_intr
18   0x86 9   PCI    Lvl Fixed  7   1     0x0/0x12  uhci_intr
19   0x83 9   PCI    Lvl Fixed  4   1     0x0/0x13  ehci_intr
20   0x87 9   PCI    Lvl Fixed  8   2     0x0/0x14  uhci_intr, uhci_intr
21   0x84 9   PCI    Lvl Fixed  5   3     0x0/0x15  uhci_intr, uhci_intr,ehci_intr
22   0x40 5   PCI    Lvl Fixed  14  1     0x0/0x16  0
32   0x20 2          Edg IPI    all 1     -         cmi_cmci_trap
56   0x82 7   PCI    Edg MSI    2   1     -         pcieb_intr_handler
57   0x30 4   PCI    Edg MSI    3   1     -         pcieb_intr_handler
58   0x60 6   PCI    Edg MSI-X  13  1     -         bnx_intr_1lvl
59   0x41 5   PCI    Lvl Fixed  15  1     0x1/0x0   drsas_isr
60   0x61 6   PCI    Edg MSI-X  0   1     -         bnx_intr_1lvl
160  0xa0 0          Edg IPI    all 0     -         poke_cpu
208  0xd0 14         Edg IPI    all 1     -         kcpc_hw_overflow_intr
209  0xd1 14         Edg IPI    all 1     -         cbe_fire
210  0xd3 14         Edg IPI    all 1     -         cbe_fire
240  0xe0 15         Edg IPI    all 1     -         xc_serv
241  0xe1 15         Edg IPI    all 1     -         apic_error_intr
...

Interrupt monitoring

[root@node01 ~] intrstat

  device |      cpu0 %tim      cpu1 %tim      cpu2 %tim      cpu3 %tim      cpu4 %tim      cpu5 %tim
-------------+------------------------------------------------------------------------------------------
   bnx#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
   bnx#1 |        24  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
  ehci#0 |         0  0.0         0  0.0         0  0.0         0  0.0         1  0.0         0  0.0
  ehci#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         6  0.0
  uhci#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
  uhci#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
  uhci#2 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         6  0.0
  uhci#3 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
  uhci#4 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         6  0.0
  uhci#5 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0