Solaris 10: Measuring Performance Historically

Solaris 10: Measuring Performance Historically

Abstract:
Computing systems have traditionally provided way to metric the health of the system. UNIX System V systems have depended upon "System Activity Reporting" or "sar" tool. The "sar" tools can be set up for automatic collection.

Reporting in Real Time:
The "sar" can be used, without scheduling, to pull data in near-real-time from the kernel by specifying an interval and an average time. One can poll the run queue statistics 5 times on 2 second intervals using "sar" with the "-q" option:

Ultra60/root# sar -q 2 5
SunOS Ultra60 5.10 Generic_141444-09 sun4u 12/06/2009
22:36:04 runq-sz %runocc swpq-sz %swpocc
22:36:06 . . 0.0 . . 0 . . . 0.0 . . . 0
22:36:08 . . 1.0 . . 50 .. . 0.0 . . . 0
22:36:10 . . 0.0 . . 0 . . . 0.0 . . . 0
22:36:12 . . 0.0 . . 0 . . . 0.0 . . . 0
22:36:14 . . 0.0 . . 0 . . . 0.0 . . . 0
Average. . . 1.0 . . 10 .. . 0.0 . . . 0

Scheduling:
Scheduling in Solaris is done using the "crontab" facility. The "cron" daemon wakes up on a regular basis and runs scheduled tasks for individual users. To see the scheduler running, it appears in the process table.

Ultra60/root# ps -elf | grep cron
0 S root 307 1 0 40 20 ? 693 ? 13:44:11 ? 0:00 /usr/sbin/cron

The task lists scheduled by users can be browsed.

Ultra2/root$ cd /var/spool/cron/crontabs
Ultra2/root$ ls -al *
-rw------- 1 root sys. 190 Sep 3 14:22 adm
-r-------- 1 root root 452 Sep 3 14:22 lp
-rw------- 1 root root 531 Dec 6 01:13 root
-rw------- 1 root sys. 308 Sep 3 14:22 sys
-r-------- 1 root sys. 404 Dec 5 06:26 uucp

Scheduling System Activity Reporting:
The "sar" is typically scheduled by the "sys" user. The default is to not run it, by commenting out sample entries.

Ultra2/root$ cd /var/spool/cron/crontabs
Ultra2/root$ cat sys
#ident "@(#)sys 1.5 92/07/14 SMI" /* SVr4.0 1.2 */
#
# The sys crontab should be used to do performance collection. See cron
# and performance manual pages for details on startup.
#
# 0 * * * 0-6 /usr/lib/sa/sa1
# 20,40 8-17 * * 1-5 /usr/lib/sa/sa1
# 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A

The following "sys" "crontab" entry will schedule 15 minute collections of performance metrics.

00,15,30,45 * * * * /usr/lib/sa/sa1

Viewing Scheduling by User:
The correct way to list your scheduling information by user is to use the "cron" with "-l" option.

Ultra60/root# crontab -l sys
#ident "@(#)sys 1.5 92/07/14 SMI" /* SVr4.0 1.2 */
#
# The sys crontab should be used to do performance collection. See cron
# and performance manual pages for details on startup.
#
# 0 * * * 0-6 /usr/lib/sa/sa1
# 20,40 8-17 * * 1-5 /usr/lib/sa/sa1
# 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A
#
00,15,30,45 * * * * /usr/lib/sa/sa1

Historic Data:
The historic data is held in a file system directory. They are stored by numeric day number for a total of one month.

Ultra60/root# cd /var/adm/sa
Ultra60/root# ls -al
total 67068
drwxrwxr-x 2 adm. sys. ....512 Dec 6 00:00 .
drwxrwxr-x 9 root sys. ....512 Dec 5 03:10 ..
-rw-r--r-- 1 sys. sys. 1290144 Dec 1 23:45 sa01
-rw-r--r-- 1 sys. sys. 1177344 Dec 2 23:45 sa02
-rw-r--r-- 1 sys. sys. 1177344 Dec 3 23:45 sa03
-rw-r--r-- 1 sys. sys. 1177344 Dec 4 23:45 sa04
-rw-r--r-- 1 sys. sys. 1177344 Dec 5 23:45 sa05
-rw-r--r-- 1 sys. sys. 1091496 Dec 6 22:00 sa06
-rw-r--r-- 1 root root ..12024 Nov 7 03:15 sa07
-rw-r--r-- 1 sys. sys. .429984 Nov 8 23:45 sa08
-rw-r--r-- 1 sys. sys. 1154304 Nov 9 23:45 sa09
-rw-r--r-- 1 sys. sys. 1154304 Nov 10 23:45 sa10
-rw-r--r-- 1 sys. sys. 1154304 Nov 11 23:45 sa11
-rw-r--r-- 1 sys. sys. 1154304 Nov 12 23:45 sa12
-rw-r--r-- 1 sys. sys. 1154304 Nov 13 23:45 sa13
-rw-r--r-- 1 sys. sys. 1154304 Nov 14 23:45 sa14
-rw-r--r-- 1 sys. sys. 1154304 Nov 15 23:45 sa15
-rw-r--r-- 1 sys. sys. 1154304 Nov 16 23:45 sa16
-rw-r--r-- 1 sys. sys. 1154304 Nov 17 23:45 sa17
-rw-r--r-- 1 sys. sys. 1154304 Nov 18 23:45 sa18
-rw-r--r-- 1 sys. sys. 1154304 Nov 19 23:45 sa19
-rw-r--r-- 1 sys. sys. 1154304 Nov 20 23:45 sa20
-rw-r--r-- 1 sys. sys. 1142280 Nov 21 23:45 sa21
-rw-r--r-- 1 sys. sys. 1173672 Nov 22 23:45 sa22
-rw-r--r-- 1 sys. sys. 1292544 Nov 23 23:45 sa23
-rw-r--r-- 1 sys. sys. 1292544 Nov 24 23:45 sa24
-rw-r--r-- 1 sys. sys. 1292544 Nov 25 23:45 sa25
-rw-r--r-- 1 sys. sys. 1292544 Nov 26 23:45 sa26
-rw-r--r-- 1 sys. sys. 1292544 Nov 27 23:45 sa27
-rw-r--r-- 1 sys. sys. 1292544 Nov 28 23:45 sa28
-rw-r--r-- 1 sys. sys. 1292544 Nov 29 23:45 sa29
-rw-r--r-- 1 sys. sys. 1292544 Nov 30 23:45 sa30

Reviewing Scheduled Data:
There are dozens of reports which can be viewed.

The historic CPU report can be seen with no option or "-u" with "sar", for the same day.

Ultra60/root# sar
SunOS Ultra60 5.10 Generic_141444-09 sun4u 12/06/2009
00:00:00 %usr %sys %wio %idle
00:15:01 0 2 0 98
00:30:00 0 2 0 98
00:45:00 0 2 0 98
01:00:00 0 2 0 98
01:15:00 0 2 0 98
...
21:15:00 0 1 0 99
21:30:01 0 1 0 99
21:45:00 0 1 0 99
22:00:00 0 1 0 99
Average 33 33 0 33

Historic memory usage can also be seen via "sar", using the "-r" flag.

Ultra60/root# sar -r
SunOS Ultra60 5.10 Generic_141444-09 sun4u 12/06/2009
00:00:00 freemem freeswap
00:15:01 163651 10451488
00:30:00 163651 10451488
00:45:00 163651 10451485
01:00:00 163651 10451485
01:15:00 163385 10443984
...
21:00:00 190656 19398416
21:15:00 190656 19398416
21:30:01 190656 19398415
21:45:00 190656 19398416
22:00:00 190656 19398416
Average. 177153 14924952

The "sar" command will also accept a file specifying a historic database, from a previous day in the month.

Ultra60/root# sar -k -f /var/adm/sa/sa02
SunOS Ultra60 5.10 Generic_141444-09 sun4u 12/02/2009
00:00:00 sml_mem. alloc. fail lg_mem... alloc fail ovsz_alloc fail
00:15:00 16646400 12855551 0 118005760 92904032 0 37765120 0
00:30:00 16646400 12858815 0 118005760 92904344 0 37765120 0
00:45:00 16646400 12855735 0 118013952 92900872 0 37765120 0
...
23:00:01 17096960 13002143 0 118767616 93431584 0 37765120 0
23:15:00 17096960 13004855 0 118767616 93433176 0 37765120 0
23:30:00 17096960 13011279 0 118775808 93434304 0 37765120 0
23:45:00 17096960 13005679 0 118775808 93428696 0 37765120 0
Average. 17031338 12977818 0 118666032 93366984 0 37765120 0

There are many other performance data sets which can be extracted once retained automatically from solaris, these are only starting examples.