Solaris 11.4 - Oracle ZFS Buffer Cache and App Memory Hints

 

Solaris 11.4 - Oracle ZFS Buffer Cache and App Memory Hints

Abstract:

Solaris has long used the UFS File System as the default for root disks. This filesystem placed I/O buffers in the "free space", causing misleading performance problems for Disk I/O when there was seemingly plenty of memory left. This was remedied by ZFS, where disk I/O buffers are counted as used buffer, truly showing only "wasted" memory as free memory. This causes a problem for some applications (i.e. Oracle RDBMS Installers), which attempt to determine how much free space is available before performing an application installation. There is a "work around" for these broken applications.

Background:

An excellent document to understand the use of memory is available from Oracle:
Memory Management Between ZFS and Applications in Oracle Solaris 11.x (Doc ID 1663862.1)

It is not the objective of this blog to re-explain this document, merely show what has not been included by Oracle, for manually making adjustments to the memory usage.

Determining Memory Usage:

One can determine the memory available for any version of Solaris with:

sun9999/root# prtconf | nawk '/Memory/'
Memory size: 32768 Megabytes

And determine the usage by different aspects of Solaris with:

sun9999/root# echo "::memstat" | mdb -k

Usage Type/Subtype                      Pages    Bytes  %Tot  %Tot/%Subt
---------------------------- ---------------- -------- ----- -----------
Kernel                                 585907    4.47g 13.9%
  Regular Kernel                       432874     3.3g       10.3%/73.8%
  Defdump prealloc                     153033    1.17g        3.6%/26.1%
ZFS                                   3063918    23.4g 73.0%
User/Anon                              269262    2.05g  6.4%
Exec and libs                            4369    34.1m  0.1%
Page Cache                              20525     160m  0.4%
Free (cachelist)                        14076     110m  0.3%
Free                                   235161    1.79g  5.6%
  Regular Free                         232151    1.77g        5.5%/98.7%
  Physical pool reserved mem             3010    23.5m        0.0%/ 1.2%
Total                                 4194304      32g  100%

Note: 5.6% is wasted memory, out of the 32Gig total, but some 23.4G /73% can be used, if requested.

The application must request the memory, for it to be available, if it can not interrogate now much it can request from the kernel.

What to Do if an App Fails to Start

Option 1: Reboot

If an application fails to start, like an Oracle RDBMS Installer, because it is not smart enough to determine most of the memory is free, you can temporarily flush the ZFS Adaptive Read Cache (ARC) by rebooting the boot.

The system will flush the cache and there will be plenty of memory available for the installer.

Option 2: Dynamically Tune

Oracle Solaris 11.4 created a new tunable called  "user_reserve_hint_pct", which requests the OS to dynamically reserve a percentage of the memory for the application by flushing the ARC.

There is a danger: do not reserve more memory for the user processes than the Kernel & OS require!

Step 1: Determine default value (should be 0)

sun9999/root# echo "user_reserve_hint_pct/D" | mdb -k
user_reserve_hint_pct:
user_reserve_hint_pct:          0

 Step 2: Select value from determined memory usage (ZFS 73 - Free 5.6 = ~67%, round down 65%)

sun9999/root# echo "user_reserve_hint_pct/W0t65" | mdb -k
user_reserve_hint_pct:
user_reserve_hint_pct:          65

sun9999/root# echo "::memstat" | mdb -k

Usage Type/Subtype                      Pages    Bytes  %Tot  %Tot/%Subt
---------------------------- ---------------- -------- ----- -----------
Kernel                                 599572    4.57g 14.2%
  Regular Kernel                       446539    3.41g       10.6%/74.4%
  Defdump prealloc                     153033    1.17g        3.6%/25.5%
ZFS                                    547332    4.18g 13.0%
User/Anon                              268734    2.05g  6.4%
Exec and libs                            4359    34.1m  0.1%
Page Cache                              20190     158m  0.4%
Free (cachelist)                        13903     109m  0.3%
Free                                  2739134    20.9g 65.3%
  Regular Free                        2736118    20.9g       65.2%/99.8%
  Physical pool reserved mem             3016    23.6m        0.0%/ 0.1%
Total                                 4194304      32g  100%

Step 3: Revert Value to Default since much of the the ARC cache has been flushed.

sun9999/root# echo "user_reserve_hint_pct/W0t0" | mdb -kw
user_reserve_hint_pct:          0x41            =       0x0

sun9999/root# echo "user_reserve_hint_pct/D" | mdb -k
user_reserve_hint_pct:
user_reserve_hint_pct:          0

At this point, the application should be able to be dynamically tuned.

Conclusion:

Oracle Solaris provides an incredibly tunable environment for running applications, including legacy applications, which are not quite smart enough to understand the benefits of the underlying OS.

Coming Soon: SPARC64 Solaris in 2021

Coming Soon: SPARC64 Solaris in 2021

Fujitsu is best described by their own hand:

The development of Fujitsu SPARC64 processors began in the early 1990s. The first generation SPARC64 ran at just 118MHz, and 20 years later SPARC64 is now approaching 4GHz; more than 30x improvement in the clock frequency alone. Fujitsu continues to enhance the technical capabilities and performance of processors in Fujitsu mainframe computers, supercomputers, and UNIX servers."

 

 

[SPARC Trademark, courtest sparc.org]

Running Fast

SPARC had been topping the charts in overall CPU performance, for years. Besides fastest raw clock rates, with the fastest threads, fastest CPU cores, fastest CPU sockets, fastest encryption, fastest decryption... capabilities also include native Oracle RDBMS Number capabilities as well as query accelerators in silicon, and security in silicon where software is restricted from snooping outside their memory areas (which Intel and other processors had been experiencing security patch slow-downs in order to deal with malware.)

 

[SPARC64 Roadmap, Courtesy fujitsu.com as of 2020-10-28]

Enhanced M12

Today, the SPARC64 M12 is one of the fastest processors on the planet, and was surpassed briefly only by the Oracle SPARC M8 processor, which arrived shortly afterwards. On-tap is what this author is describing as the SPARC64 M12+, for 2021.

The M12 maxed out at 3TB or RAM in the 2 processor chassis, while the next generation should be 1.5x larger - this author is hoping for a 4TB memory footprint in a 2x CPU chassis. Previous chassis supported Solaris 10, Solaris 11 should be supported at the minimum. This author is expecting a Solaris 11.5 release.