Installing a ISO from ILOM 4.0.3+ using SSH

Abstract:

The SPARC platform have long come with various Lights Out Management (LOM) capabilities, to access the hardware, and provide for access to the OS from underneath, when there is a hardware issue. A more advanced system called Integrated Lights Out Management (ILOM) was later created. With ILOM 4.0.3, a feature was created to allow for the boot from a remove ISO via SSH!

Where to get ISO:

The easiest place to get the most recent version of Solaris, such as the Common Build Edition

https://www.oracle.com/solaris/solaris11/downloads/solaris-downloads.html

Various ISO's are available from Oracle for Solaris:

Where to download Oracle Solaris ISO images and Update Releases (Doc ID 1277964.1) 
https://support.oracle.com/epmos/faces/DocumentDisplay?id=1277964.1

What to do from ILOM:

Copy ISO's to a local directory on a server

a.b.c.d/user$ ls -l /export/home/user/*iso

a.b.c.d/user$ ls -al /u000/P2V/iso/*iso

-rw-r--r--   1 user root   2314731520 May 12  2016 /export/home/user/sol-10-u11-ga-sparc-dvd.iso

-rw-r--r--   1 user root     867020800 May 13  2016 /export/home/user/sol-11_3-text-sparc.iso

-rw-r--r--   1 user root   1018736640 Apr 23  2019 /export/home/user/sol-11_4-text-sparc.iso

-rw-r--r--   1 user root     551464960 Oct 20  2011 /export/home/user/sol-11-1111-text-sparc.iso

Make sure there is connectivity from the ILOM to the Server hosting the ISO

-> set /SP/network/test ping=a.b.c.d
Ping of a.b.c.d succeeded

Set the ILOM Host Storage Device to Remote

-> set /SP/services/kvms/host_storage_device/ mode=remote

Set the username, password, and ISO location

-> cd /SP/services/kvms/host_storage_device/remote

-> set username=user
-> set password=password
-> set server_URI=sshfs://a.b.c.d:/export/home/user/sol-11_4-text-sparc.iso

/SP/services/kvms/host_storage_device=remote
Targets:
Properties:
password = *****
server_URI = sshfs://a.b.c.d:/export/home/user/sol-11_4-text-sparc.iso
username = user

Review Values

-> show /SP/services/kvms/host_storage_device/

/SP/services/kvms/host_storage_device

Targets:
remote

Properties:
mode = remote
status = operational 

Stop Automatic Boot on Host

-> set /HOST/bootmode script="setenv auto-boot? false"

Mount & Boot the Remote ISO

-> start /SP/console -script 

{ok} reset-all
{ok} devalias
...
rcdrom

{ok} ok boot rcdrom

Boot device: /pci@311/pci@1/usb@0/storage@1/disk@0 File and args:
SunOS Release 5.11 Version 11.4.0.15.0 64-bit
Copyright (c) 1983, 2018, Oracle and/or its affiliates. All rights reserved.
Remounting root read/write
Probing for device nodes ...
Preparing image for use
NOTICE: mount: not a UFS magic number (0x0)
NOTICE: mount: not a UFS magic number (0x0)
Done mounting image
USB keyboard
1. Arabic 15. Korean
2. Belgian 16. Latin-American
3. Brazilian 17. Norwegian
4. Canadian-Bilingual 18. Portuguese
5. Canadian-French 19. Russian
6. Danish 20. Spanish
7. Dutch 21. Swedish
8. Dvorak 22. Swiss-French
9. Finnish 23. Swiss-German
10. French 24. Traditional-Chinese
11. German 25. TurkishQ
12. Italian 26. UK-English
13. Japanese-type6 27. US-English
14. Japanese

To select the keyboard layout, enter a number [default 27]:

Additional Information

A good note on this process is available for people with Oracle Support:
How to Install/Re-image a T5-x, S7, T7-x, T8-x, M7-x, or M8-x System Using the sshfs Protocol (Doc ID 2817892.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2817892.1

Oracle 19c Installer: Root Equivalence Fails

 

Oracle 19c Installer: Root Equivalence Fails

Abstract:

The Oracle Installer is a common component for interactions with Oracle databases. It has a tendency of being a little buggy, very sensitive to the underlying operating system. When Solaris 11.4 upgraded, some of the underlying components are no longer compatible with the 19c installer, so workarounds must be implemented.

[Oracle RAC Architecture, courtesy Oracle Tutorial]

Oracle Architecture

When the installer is setting up the database in a RAC cluster, there is a procedure to set up root equivalence or test root equivalence. This is essentially password-less ssh between clustered nodes The process performs an "scp" of a file between the clustered hosts, and this can fail.

What can possibly go wrong?

A complete list of common failures and workarounds for the installer is located in an Oracle Note:
TOP Note: Solutions for Typical Grid Infrastructure/RAC Database runInstaller Issues (Doc ID 1056713.1)

The Oracle19c installer does not do "strict filename checking", which makes it incompatible with OpenSSH 8.x and newer since Versions 8.x and above enable “strict filename checking” by default. 

See Oracle Doc ID 2555697.1

Does it apply to my situation?

The scp binary must be wrapped with a script that calls the binary with a special compatibility flag.
(Note: fix will be undone as future “ssh” patches are applied and the workaround will need repeating if installer is needed in the future... which is why the procedure I provide below is important, so the wrapper script does not get purged during an upgrade.)

Check version of ssh to determine if system Oracle 19c is installing against is too new.

sun2202/oracle$ ssh -V
OpenSSH_8.1p1, OpenSSL 1.0.2u  20 Dec 2019

Note: The ssh software is too new for the Oracle 19c installer

Verify ssh is a binary and the workaround / fix has not already been implemented.

sun9999/oracle$ ls -alt /usr/bin/scp
-r-xr-xr-x   1 root     bin       135864 Jan  6 02:49 /usr/bin/scp

sun9999/oracle$ file /usr/bin/scp
/usr/bin/scp:       ELF 64-bit MSB dynamic lib SPARCV9 Version 1, UltraSPARC3 Extensions Required, position-independent executable, dynamically linked, not stripped, no debugging information available

Note: The scp command appears to be an original OS binary, meaning a wrapper can be applied.

What is the work around?

If the old Oracle 19c Installer is used, create shell wrapper to disable “strict filename checking.”

Create the wrapper and check permissions & ownership.

sun9999/root# cat /usr/bin/scp.Doc.ID.2555697.1

#
# bug in oracle installer, for compatibility with OpenSSH 8.x
# INS-06006 GI RunInstaller Fails If OpenSSH Is Upgraded to 8.x
# (Doc ID 2555697.1)
#
/usr/bin/scp.orig -T $* 

sun9999/root# ls -al /usr/bin/scp.Doc.ID.2555697.1
-r-xr-xr-x 1 root  bin   209 Jun 15  2020 /usr/bin/scp.Doc.ID.2555697.1

Note: the shell wrapper above was created, disables checking, and has corret permissions & ownership.

Show Binaries & Shell Wrapper

sun9999/root# ls -alt /usr/bin/scp*
-r-xr-xr-x 1 root bin 135864 Jan  6 02:49 /usr/bin/scp
-r-xr-xr-x 1 root bin    209 Jun 15  2020 /usr/bin/scp.Doc.ID.2555697.1

Copy binary to “.orig” for Wrapper, Move binary to backup [by OS patch], and Copy Wrapper in place.

sun9999/root# uname -a
SunOS sun2202 5.11 11.4.28.82.3 sun4v sparc sun4v

sun9999/oracle$ Backup=/usr/bin/scp.11.4.28

sun9999/oracle$ echo $Backup
/usr/bin/scp.11.4.28

sun9999/root# cp -p  /usr/bin/scp /usr/bin/scp.orig
sun9999/root# mv     /usr/bin/scp $Backup
sun9999/root# scp -p /usr/bin/scp.Doc.ID.2555697.1 /usr/bin/scp

sun9999/root# chown root:bin     /usr/bin/scp
sun9999/root# chmod 555          /usr/bin/scp

Show Binaries & Shell Wrapper

sun9999/root# ls -alt /usr/bin/scp*
-r-xr-xr-x 1 root bin    209 Jan 19 16:23 /usr/bin/scp
-r-xr-xr-x 1 root bin 135864 Jan  6 02:49 /usr/bin/scp.11.4.28
-r-xr-xr-x 1 root bin 135864 Jan  6 02:49 /usr/bin/scp.orig
-r-xr-xr-x 1 root bin    209 Jun 15  2020 /usr/bin/scp.Doc.ID.2555697.1

Verify scp script is functional

sun9999/oracle$ type scp
scp is hashed (/usr/bin/scp)

sun9999/oracle$ scp
usage: scp [-346BCpqrTv] [-c cipher] [-F ssh_config] [-i identity_file]
            [-J destination] [-l limit] [-o ssh_option] [-P port]
            [-S program] source ... target

Instruct the DBA’s to resume use of the Oracle 19c installer

Caveats:

When an upgrade happens, it will be important to identify if the "scp" command is no longer a script and has been reverted to the binary.

If this had occurred, follow the same steps above:

1. creating a new ".orig"
2. create a new backup of the binary tagging it by what OS release & SRU
3. copy the wrapper back into place, with proper ownership & permissions

With the old wrapper & backups in place, you should be able to figure out what needs to be done without trying to find these instructions again.