CPU/RAM Troubleshooting¶
How to display the amount of logical processors¶
[root@node01 ~] psrinfo -vp
The physical processor has 4 cores and 8 virtual processors (1 3 5 7 9 11 13 15)
The core has 2 virtual processors (1 9)
The core has 2 virtual processors (3 11)
The core has 2 virtual processors (5 13)
The core has 2 virtual processors (7 15)
x86 (GenuineIntel 106A5 family 6 model 26 step 5 clock 2267 MHz)
Intel(r) Xeon(r) CPU E5520 @ 2.27GHz
The physical processor has 4 cores and 8 virtual processors (0 2 4 6 8 10 12 14)
The core has 2 virtual processors (0 8)
The core has 2 virtual processors (2 10)
The core has 2 virtual processors (4 12)
The core has 2 virtual processors (6 14)
x86 (GenuineIntel 106A5 family 6 model 26 step 5 clock 2267 MHz)
Intel(r) Xeon(r) CPU E5520 @ 2.27GHz
How to display CPU statistics and load distribution among processors¶
[root@node01 ~] mpstat 1 5
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 870 1 274 2440 235 569 2 47 59 0 4709 2 3 0 94
1 547 1 58 203 63 462 3 19 35 0 4964 2 2 0 95
2 326 0 142 424 132 690 1 29 52 0 2427 1 2 0 97
3 168 1 38 325 123 814 1 17 39 0 1875 1 2 0 97
4 105 0 31 326 135 224 0 16 37 0 1060 1 1 0 98
5 114 0 25 216 87 382 1 16 31 0 1192 1 1 0 98
6 116 0 38 459 153 769 1 14 37 0 2732 1 2 0 97
7 241 1 20 656 185 762 1 15 33 0 1466 0 1 0 98
8 95 0 34 230 92 327 1 9 26 0 1620 1 1 0 98
9 111 0 18 238 86 691 1 15 34 0 1481 1 1 0 98
10 66 0 37 160 64 219 0 7 25 0 1184 0 1 0 99
11 103 0 21 157 61 345 1 12 26 0 1281 0 1 0 99
12 50 0 20 721 194 306 1 10 28 0 1318 0 1 0 98
13 129 0 203 320 141 741 1 15 33 0 1445 1 1 0 98
14 34 0 13 213 72 342 0 7 25 0 1453 0 1 0 99
15 94 0 20 406 226 367 1 12 54 0 883 0 1 0 98
- CPU - Processor ID.
Note
A high number of following calls can be an indication of performance problems.
- xcal - Number of inter-processor cross-calls.
- intr - Number of interrupts.
- smtx - Number of operations, which couldn’t acquire a lock on the first try (spins on mutexes).
- syscl - Number of syscalls.
- usr - Percent of processor user time.
- sys - Percent of processor system time.
- idl - Percent of processor idle time.
How to display the amount of total memory available in the system (MiB)¶
[root@node01 ~] prtconf -m
32758
How to display virtual memory usage¶
[root@node01 ~] vmstat
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr lf rm s0 s1 in sy cs us sy id
0 0 0 54267716 25895544 366 3131 74 0 0 0 240 6 -268 186 -1 7442 30957 7950 1 1 98
- r - The number of kernel threads in run queue. High values of r may indicate performance issues.
- us - User time (%).
- sy - System time (%).
- id - Idle time (%).
How to display processes on the system and in the zones¶
[root@node01 ~] prstat -Z
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
3538 root 2095M 2083M sleep 1 0 0:03:54 0.4% qemu-system-x86/12
3421 root 2094M 2082M sleep 59 0 0:01:09 0.1% qemu-system-x86/5
90 root 0K 0K sleep 99 -20 0:00:18 0.0% zpool-zones/238
3601 root 47M 31M sleep 59 0 0:00:07 0.0% python2.7/1
3585 root 38M 28M sleep 59 0 0:00:03 0.0% python2.7/1
3469 root 37M 28M sleep 59 0 0:00:03 0.0% python2.7/1
2843 root 14M 11M sleep 1 0 0:00:03 0.0% kvmiostat/1
3632 root 42M 17M sleep 1 0 0:00:00 0.0% python2.7/1
9573 root 4560K 3408K cpu1 1 0 0:00:00 0.0% prstat/1
184 root 6156K 2948K sleep 1 0 0:00:00 0.0% syseventd/18
2829 root 3996K 2436K sleep 1 0 0:00:00 0.0% vm-kvm-disk-io-/1
2806 root 6092K 4600K sleep 1 0 0:00:00 0.0% vm-network-moni/1
2203 root 2604K 812K sleep 29 0 0:00:00 0.0% lldpd/1
2801 root 1876K 1180K sleep 59 0 0:00:00 0.0% ctrun/1
2738 root 3952K 892K sleep 59 0 0:00:00 0.0% ipmon/1
2709 root 4888K 3560K sleep 1 0 0:00:00 0.0% picld/4
2821 root 2120K 1084K sleep 59 0 0:00:00 0.0% cron/1
2852 daemon 4116K 1812K sleep 1 0 0:00:00 0.0% rpcbind/3
3560 root 35M 16M sleep 1 0 0:00:00 0.0% python2.7/1
2834 root 5144K 1216K sleep 59 0 0:00:00 0.0% zabbix_agentd/1
88 root 2664K 1524K sleep 29 0 0:00:00 0.0% pfexecd/3
21 root 2988K 1608K sleep 29 0 0:00:00 0.0% dlmgmtd/7
2825 root 4868K 3392K sleep 1 0 0:00:00 0.0% vm-cpu-monitor/1
2835 root 5144K 1216K sleep 59 0 0:00:00 0.0% zabbix_agentd/1
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
3 2 2095M 2083M 6.4% 0:03:54 0.4% f7860689-c435-4964-9f7d-2d2*
1 2 2094M 2082M 6.4% 0:01:09 0.1% a28faa4d-d0ee-4593-938a-f0d*
0 89 1330M 629M 1.8% 0:00:50 0.1% global
Total: 93 processes, 521 lwps, load averages: 0.25, 0.26, 0.26
PID - Process ID.
USERNAME - The real user (login) name or real user ID.
SIZE - The total virtual memory size of the process, including all mapped files and devices.
RSS - The resident set size of the process (RSS), in kilobytes (K), megabytes (M), or gigabytes (G).
STATE - The state of the process.
- cpuN - Process is running on CPU
N. - sleep - Sleeping: process is waiting for an event to complete.
- wait - Waiting: process is waiting for CPU usage to drop to the CPU-caps enforced limit.
- run - Runnable: process in on run queue.
- zombie - Zombie state: process terminated and parent not waiting.
- stop - Process is stopped.
- cpuN - Process is running on CPU
TIME - The cumulative execution time for the process.
CPU - The percentage of recent CPU time used by the process.
PROCESS - The name of the process.
How to display memory usage of virtual machines¶
[root@node01 ~] zonememstat
ZONE RSS(MB) CAP(MB) NOVER POUT(MB)
global 0 - - -
a28faa4d-d0ee-4593-938a-f0d062022b02 2082 3072 0 0
f7860689-c435-4964-9f7d-2d2d70cfe389 2082 3072 0 0
- ZONE - The zone name.
- RSS - The amount of physical memory consumed.
- CAP - The memory cap.
- NOVER - Number of times the zone reached over its cap.
- POUT - The amount of paged out memory.
How to display memory usage by type¶
[root@node01 ~] echo ::memstat | mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 462970 1808 6%
ZFS File Data 127036 496 2%
Anon 1169355 4567 14%
Exec and libs 4516 17 0%
Page cache 15842 61 0%
Free (cachelist) 19856 77 0%
Free (freelist) 6584225 25719 79%
Total 8383800 32749
Physical 8383799 32749
How to display the amount of consumed memory by processes and virtual machines¶
[root@node01 ~] prstat -s rss -z 3
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
3538 root 2095M 2083M sleep 1 0 0:04:11 0.5% qemu-system-x86/12
3274 root 0K 0K sleep 60 - 0:00:00 0.0% zsched/1
Total: 2 processes, 13 lwps, load averages: 0.26, 0.26, 0.26
Interrupt mapping¶
[root@node01 ~] echo ::interrupts | mdb -k
IRQ Vect IPL Bus Trg Type CPU Share APIC/INT# ISR(s)
3 0xb1 12 ISA Edg Fixed 10 1 0x0/0x3 asyintr
4 0xb0 12 ISA Edg Fixed 9 1 0x0/0x4 asyintr
9 0x81 9 PCI Lvl Fixed 1 1 0x0/0x9 acpi_wrapper_isr
17 0x85 9 PCI Lvl Fixed 6 1 0x0/0x11 uhci_intr
18 0x86 9 PCI Lvl Fixed 7 1 0x0/0x12 uhci_intr
19 0x83 9 PCI Lvl Fixed 4 1 0x0/0x13 ehci_intr
20 0x87 9 PCI Lvl Fixed 8 2 0x0/0x14 uhci_intr, uhci_intr
21 0x84 9 PCI Lvl Fixed 5 3 0x0/0x15 uhci_intr, uhci_intr,ehci_intr
22 0x40 5 PCI Lvl Fixed 14 1 0x0/0x16 0
32 0x20 2 Edg IPI all 1 - cmi_cmci_trap
56 0x82 7 PCI Edg MSI 2 1 - pcieb_intr_handler
57 0x30 4 PCI Edg MSI 3 1 - pcieb_intr_handler
58 0x60 6 PCI Edg MSI-X 13 1 - bnx_intr_1lvl
59 0x41 5 PCI Lvl Fixed 15 1 0x1/0x0 drsas_isr
60 0x61 6 PCI Edg MSI-X 0 1 - bnx_intr_1lvl
160 0xa0 0 Edg IPI all 0 - poke_cpu
208 0xd0 14 Edg IPI all 1 - kcpc_hw_overflow_intr
209 0xd1 14 Edg IPI all 1 - cbe_fire
210 0xd3 14 Edg IPI all 1 - cbe_fire
240 0xe0 15 Edg IPI all 1 - xc_serv
241 0xe1 15 Edg IPI all 1 - apic_error_intr
...
Interrupt monitoring¶
[root@node01 ~] intrstat
device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 %tim cpu4 %tim cpu5 %tim
-------------+------------------------------------------------------------------------------------------
bnx#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
bnx#1 | 24 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0 0 0.0
ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 6 0.0
uhci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
uhci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
uhci#2 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 6 0.0
uhci#3 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
uhci#4 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 6 0.0
uhci#5 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0