Oracle Database Pre-Requisites Package

Oracle Database Pre-Requisites Package

Abstract:

The Oracle RDBMS Database has long been a tool on UNIX platforms. With the acquisition of Sun Microsystems, Oracle has drawn the Solaris operating system closer to it's orbit. One such change was to make a "pre-requisites" package in Solaris 11, to simplify installation of the RDBMS. There is a caution, if you already have standardized on user & group names for database usage.

Solaris Pre-Requisites

I was not going to write anything about this, but it appears that even other skilled engineers have run into problems with the automation of the Solaris Pre-Requisites package. I had seen a blog post from Alan, who graciously provided a solution when deploying hundreds of systems automatically.

sun9876/root# pkg install oracle-database-preinstall-19c

Alan struggled with ZFS Filesystems being created for new users by the pre-requisites script, before the user filesystems were mounted. This is not the only place where ZFS Filesystems being created for users are a problem, this author experienced several other conditions, one such condition kept a production system from coming up when the id's were scrambled by another such script.

This author filed a BUG to stop creating a ZFS File System for every user created. We have hundreds of users on some of our servers, so this ZFS feature was an inhibition to moving from Solaris 10 to 11. If you enable this feature in 11.4, you might be able to avoid the "avoid" that Alan had to use, if you don't mind the user & group id's being re-created.

Not all data centers deploy hundreds of Oracle DB's simultaneously. This author kept the pre-requisites script, but the default nature "blew away" our old user & group id's, adjusted ownership, and it was a nightmare. Procedures and scripts were built to undo everything that the pre-requisites package did with the id's & home dirctory, including returning the oracle user & dba groups back to their original ID's, correcting user & group ownership in /export/home, and then correcting the passwd, shadow, and group database. What a nightmare!

Conclusions

Honestly, it should be considered a BUG to rip out user & group names if those user & group names already exist, as a package overwrites them. After doing some DBA reading (this author is a systems guy first now a days) and do what the DBA's request of me, I found they missed a step which was to perform an "avoid" in order to avoid wiping out existing user & group id's for replacement, prior the preinstallation!

sun9876/root# pkg avoid oracle-database-os-configuration

There should NEVER be the need to perform an "avoid" from doing likely harm, but rather the package should be smart enough to realize id's exist and use them as defaults... and use an "avoid" clause, or something similar, so the "check" can be avoided and selectively allow the dangerous "blowing away & replacing" behavior.

Another piece of advise is to always perform a "dry-run" of an installation, before adding a package you are not aware of.

sun9876/root# pkg install -n oracle-database-preinstall-19c 

The dry run will show details that need to be understood, before applying an actual package later.

Oracle 19c Installer: Root Equivalence Fails

 

Oracle 19c Installer: Root Equivalence Fails

Abstract:

The Oracle Installer is a common component for interactions with Oracle databases. It has a tendency of being a little buggy, very sensitive to the underlying operating system. When Solaris 11.4 upgraded, some of the underlying components are no longer compatible with the 19c installer, so workarounds must be implemented.

[Oracle RAC Architecture, courtesy Oracle Tutorial]

Oracle Architecture

When the installer is setting up the database in a RAC cluster, there is a procedure to set up root equivalence or test root equivalence. This is essentially password-less ssh between clustered nodes The process performs an "scp" of a file between the clustered hosts, and this can fail.

What can possibly go wrong?

A complete list of common failures and workarounds for the installer is located in an Oracle Note:
TOP Note: Solutions for Typical Grid Infrastructure/RAC Database runInstaller Issues (Doc ID 1056713.1)

The Oracle19c installer does not do "strict filename checking", which makes it incompatible with OpenSSH 8.x and newer since Versions 8.x and above enable “strict filename checking” by default. 

See Oracle Doc ID 2555697.1

Does it apply to my situation?

The scp binary must be wrapped with a script that calls the binary with a special compatibility flag.
(Note: fix will be undone as future “ssh” patches are applied and the workaround will need repeating if installer is needed in the future... which is why the procedure I provide below is important, so the wrapper script does not get purged during an upgrade.)

Check version of ssh to determine if system Oracle 19c is installing against is too new.

sun2202/oracle$ ssh -V
OpenSSH_8.1p1, OpenSSL 1.0.2u  20 Dec 2019

Note: The ssh software is too new for the Oracle 19c installer

Verify ssh is a binary and the workaround / fix has not already been implemented.

sun9999/oracle$ ls -alt /usr/bin/scp
-r-xr-xr-x   1 root     bin       135864 Jan  6 02:49 /usr/bin/scp

sun9999/oracle$ file /usr/bin/scp
/usr/bin/scp:       ELF 64-bit MSB dynamic lib SPARCV9 Version 1, UltraSPARC3 Extensions Required, position-independent executable, dynamically linked, not stripped, no debugging information available

Note: The scp command appears to be an original OS binary, meaning a wrapper can be applied.

What is the work around?

If the old Oracle 19c Installer is used, create shell wrapper to disable “strict filename checking.”

Create the wrapper and check permissions & ownership.

sun9999/root# cat /usr/bin/scp.Doc.ID.2555697.1

#
# bug in oracle installer, for compatibility with OpenSSH 8.x
# INS-06006 GI RunInstaller Fails If OpenSSH Is Upgraded to 8.x
# (Doc ID 2555697.1)
#
/usr/bin/scp.orig -T $* 

sun9999/root# ls -al /usr/bin/scp.Doc.ID.2555697.1
-r-xr-xr-x 1 root  bin   209 Jun 15  2020 /usr/bin/scp.Doc.ID.2555697.1

Note: the shell wrapper above was created, disables checking, and has corret permissions & ownership.

Show Binaries & Shell Wrapper

sun9999/root# ls -alt /usr/bin/scp*
-r-xr-xr-x 1 root bin 135864 Jan  6 02:49 /usr/bin/scp
-r-xr-xr-x 1 root bin    209 Jun 15  2020 /usr/bin/scp.Doc.ID.2555697.1

Copy binary to “.orig” for Wrapper, Move binary to backup [by OS patch], and Copy Wrapper in place.

sun9999/root# uname -a
SunOS sun2202 5.11 11.4.28.82.3 sun4v sparc sun4v

sun9999/oracle$ Backup=/usr/bin/scp.11.4.28

sun9999/oracle$ echo $Backup
/usr/bin/scp.11.4.28

sun9999/root# cp -p  /usr/bin/scp /usr/bin/scp.orig
sun9999/root# mv     /usr/bin/scp $Backup
sun9999/root# scp -p /usr/bin/scp.Doc.ID.2555697.1 /usr/bin/scp

sun9999/root# chown root:bin     /usr/bin/scp
sun9999/root# chmod 555          /usr/bin/scp

Show Binaries & Shell Wrapper

sun9999/root# ls -alt /usr/bin/scp*
-r-xr-xr-x 1 root bin    209 Jan 19 16:23 /usr/bin/scp
-r-xr-xr-x 1 root bin 135864 Jan  6 02:49 /usr/bin/scp.11.4.28
-r-xr-xr-x 1 root bin 135864 Jan  6 02:49 /usr/bin/scp.orig
-r-xr-xr-x 1 root bin    209 Jun 15  2020 /usr/bin/scp.Doc.ID.2555697.1

Verify scp script is functional

sun9999/oracle$ type scp
scp is hashed (/usr/bin/scp)

sun9999/oracle$ scp
usage: scp [-346BCpqrTv] [-c cipher] [-F ssh_config] [-i identity_file]
            [-J destination] [-l limit] [-o ssh_option] [-P port]
            [-S program] source ... target

Instruct the DBA’s to resume use of the Oracle 19c installer

Caveats:

When an upgrade happens, it will be important to identify if the "scp" command is no longer a script and has been reverted to the binary.

If this had occurred, follow the same steps above:

1. creating a new ".orig"
2. create a new backup of the binary tagging it by what OS release & SRU
3. copy the wrapper back into place, with proper ownership & permissions

With the old wrapper & backups in place, you should be able to figure out what needs to be done without trying to find these instructions again.

Solaris 11.4 - Oracle ZFS Buffer Cache and App Memory Hints

 

Solaris 11.4 - Oracle ZFS Buffer Cache and App Memory Hints

Abstract:

Solaris has long used the UFS File System as the default for root disks. This filesystem placed I/O buffers in the "free space", causing misleading performance problems for Disk I/O when there was seemingly plenty of memory left. This was remedied by ZFS, where disk I/O buffers are counted as used buffer, truly showing only "wasted" memory as free memory. This causes a problem for some applications (i.e. Oracle RDBMS Installers), which attempt to determine how much free space is available before performing an application installation. There is a "work around" for these broken applications.

Background:

An excellent document to understand the use of memory is available from Oracle:
Memory Management Between ZFS and Applications in Oracle Solaris 11.x (Doc ID 1663862.1)

It is not the objective of this blog to re-explain this document, merely show what has not been included by Oracle, for manually making adjustments to the memory usage.

Determining Memory Usage:

One can determine the memory available for any version of Solaris with:

sun9999/root# prtconf | nawk '/Memory/'
Memory size: 32768 Megabytes

And determine the usage by different aspects of Solaris with:

sun9999/root# echo "::memstat" | mdb -k

Usage Type/Subtype                      Pages    Bytes  %Tot  %Tot/%Subt
---------------------------- ---------------- -------- ----- -----------
Kernel                                 585907    4.47g 13.9%
  Regular Kernel                       432874     3.3g       10.3%/73.8%
  Defdump prealloc                     153033    1.17g        3.6%/26.1%
ZFS                                   3063918    23.4g 73.0%
User/Anon                              269262    2.05g  6.4%
Exec and libs                            4369    34.1m  0.1%
Page Cache                              20525     160m  0.4%
Free (cachelist)                        14076     110m  0.3%
Free                                   235161    1.79g  5.6%
  Regular Free                         232151    1.77g        5.5%/98.7%
  Physical pool reserved mem             3010    23.5m        0.0%/ 1.2%
Total                                 4194304      32g  100%

Note: 5.6% is wasted memory, out of the 32Gig total, but some 23.4G /73% can be used, if requested.

The application must request the memory, for it to be available, if it can not interrogate now much it can request from the kernel.

What to Do if an App Fails to Start

Option 1: Reboot

If an application fails to start, like an Oracle RDBMS Installer, because it is not smart enough to determine most of the memory is free, you can temporarily flush the ZFS Adaptive Read Cache (ARC) by rebooting the boot.

The system will flush the cache and there will be plenty of memory available for the installer.

Option 2: Dynamically Tune

Oracle Solaris 11.4 created a new tunable called  "user_reserve_hint_pct", which requests the OS to dynamically reserve a percentage of the memory for the application by flushing the ARC.

There is a danger: do not reserve more memory for the user processes than the Kernel & OS require!

Step 1: Determine default value (should be 0)

sun9999/root# echo "user_reserve_hint_pct/D" | mdb -k
user_reserve_hint_pct:
user_reserve_hint_pct:          0

 Step 2: Select value from determined memory usage (ZFS 73 - Free 5.6 = ~67%, round down 65%)

sun9999/root# echo "user_reserve_hint_pct/W0t65" | mdb -k
user_reserve_hint_pct:
user_reserve_hint_pct:          65

sun9999/root# echo "::memstat" | mdb -k

Usage Type/Subtype                      Pages    Bytes  %Tot  %Tot/%Subt
---------------------------- ---------------- -------- ----- -----------
Kernel                                 599572    4.57g 14.2%
  Regular Kernel                       446539    3.41g       10.6%/74.4%
  Defdump prealloc                     153033    1.17g        3.6%/25.5%
ZFS                                    547332    4.18g 13.0%
User/Anon                              268734    2.05g  6.4%
Exec and libs                            4359    34.1m  0.1%
Page Cache                              20190     158m  0.4%
Free (cachelist)                        13903     109m  0.3%
Free                                  2739134    20.9g 65.3%
  Regular Free                        2736118    20.9g       65.2%/99.8%
  Physical pool reserved mem             3016    23.6m        0.0%/ 0.1%
Total                                 4194304      32g  100%

Step 3: Revert Value to Default since much of the the ARC cache has been flushed.

sun9999/root# echo "user_reserve_hint_pct/W0t0" | mdb -kw
user_reserve_hint_pct:          0x41            =       0x0

sun9999/root# echo "user_reserve_hint_pct/D" | mdb -k
user_reserve_hint_pct:
user_reserve_hint_pct:          0

At this point, the application should be able to be dynamically tuned.

Conclusion:

Oracle Solaris provides an incredibly tunable environment for running applications, including legacy applications, which are not quite smart enough to understand the benefits of the underlying OS.