Solaris 11.4 - Oracle ZFS Buffer Cache and App Memory Hints
Abstract:
Solaris has long used the UFS File System as the default for root disks. This filesystem placed I/O buffers in the "free space", causing misleading performance problems for Disk I/O when there was seemingly plenty of memory left. This was remedied by ZFS, where disk I/O buffers are counted as used buffer, truly showing only "wasted" memory as free memory. This causes a problem for some applications (i.e. Oracle RDBMS Installers), which attempt to determine how much free space is available before performing an application installation. There is a "work around" for these broken applications.
Background:
An excellent document to understand the use of memory is available from Oracle:
Memory Management Between ZFS and Applications in Oracle Solaris 11.x (Doc ID 1663862.1)
It is not the objective of this blog to re-explain this document, merely show what has not been included by Oracle, for manually making adjustments to the memory usage.
Determining Memory Usage:
One can determine the memory available for any version of Solaris with:
sun9999/root# prtconf | nawk '/Memory/'
Memory size: 32768 Megabytes
And determine the usage by different aspects of Solaris with:
sun9999/root# echo "::memstat" | mdb -k
Usage Type/Subtype Pages Bytes
%Tot %Tot/%Subt
---------------------------- ---------------- --------
----- -----------
Kernel 585907
4.47g 13.9%
Regular Kernel 432874 3.3g
10.3%/73.8%
Defdump
prealloc 153033 1.17g
3.6%/26.1%
ZFS 3063918 23.4g 73.0%
User/Anon 269262 2.05g
6.4%
Exec and
libs 4369 34.1m
0.1%
Page Cache 20525 160m
0.4%
Free
(cachelist)
14076 110m 0.3%
Free 235161
1.79g 5.6%
Regular Free 232151 1.77g
5.5%/98.7%
Physical pool reserved mem 3010 23.5m
0.0%/ 1.2%
Total 4194304 32g
100%
Note: 5.6% is wasted memory, out of the 32Gig total, but some 23.4G /73% can be used, if requested.
The application must request the memory, for it to be available, if it can not interrogate now much it can request from the kernel.
What to Do if an App Fails to Start
Option 1: Reboot
If an application fails to start, like an Oracle RDBMS Installer, because it is not smart enough to determine most of the memory is free, you can temporarily flush the ZFS Adaptive Read Cache (ARC) by rebooting the boot.
The system will flush the cache and there will be plenty of memory available for the installer.
Option 2: Dynamically Tune
Oracle Solaris 11.4 created a new tunable called "user_reserve_hint_pct", which requests the OS to dynamically reserve a percentage of the memory for the application by flushing the ARC.
There is a danger: do not reserve more memory for the user processes than the Kernel & OS require!
Step 1: Determine default value (should be 0)
sun9999/root#
echo "user_reserve_hint_pct/D" | mdb -k
user_reserve_hint_pct:
user_reserve_hint_pct: 0
Step 2: Select value from determined memory usage (ZFS 73 - Free 5.6 = ~67%, round down 65%)
sun9999/root#
echo "user_reserve_hint_pct/W0t65" | mdb -k
user_reserve_hint_pct:
user_reserve_hint_pct: 65
sun9999/root# echo "::memstat" | mdb -k
Usage Type/Subtype Pages Bytes
%Tot %Tot/%Subt
---------------------------- ---------------- --------
----- -----------
Kernel
599572 4.57g 14.2%
Regular Kernel 446539 3.41g
10.6%/74.4%
Defdump
prealloc 153033 1.17g
3.6%/25.5%
ZFS 547332 4.18g 13.0%
User/Anon 268734 2.05g
6.4%
Exec and
libs 4359 34.1m
0.1%
Page Cache 20190 158m
0.4%
Free
(cachelist)
13903 109m 0.3%
Free 2739134 20.9g 65.3%
Regular Free 2736118 20.9g
65.2%/99.8%
Physical pool reserved mem 3016 23.6m
0.0%/ 0.1%
Total 4194304 32g
100%
Step 3: Revert Value to Default since much of the the ARC cache has been flushed.
sun9999/root# echo "user_reserve_hint_pct/W0t0" | mdb -kw
user_reserve_hint_pct: 0x41 = 0x0
sun9999/root# echo "user_reserve_hint_pct/D" | mdb
-k
user_reserve_hint_pct:
user_reserve_hint_pct: 0
At this point, the application should be able to be dynamically tuned.
Conclusion:
Oracle Solaris provides an incredibly tunable environment for running applications, including legacy applications, which are not quite smart enough to understand the benefits of the underlying OS.
