OpsCenter 12c 12.4, Patches, and Solaris 11.4 SRU 48

 

OpsCenter 12c 12.4, Patches, and Solaris 11.4 SRU 48

Abstract

Sun Microsystems purchased a company, which performed data center management, across multiple platforms, and then decided to merge it into Solaris. The Sun Connect product was born, to help deliver patches more effectively. The N1 Compute Initiative was born, to treat all systems in the data center as a single entity. OpsCenter was made available for all customers, to do local provisioning, patch, health, and reporting. Oracle purchased Sun Microsystems and had their own management framework called Oracle Enterprise Manager, but it was always short on handling hardware through provisioning hardware & management of the ILOM hardware [without an OS]. Oracle refers to OpsCenter as Oracle Enterprise Manager OpsCenter. 

Recent History

Oracle had been updating OpC pretty aggressively in 2022, this article talks about the path to accomplish this. The first set of updates was associated with Log4J vulnerability in December 2021, but introduction of Oracle Solaris 11.4 SRU 48 on agents actually caused a disconnection to the OpsCenter server, and some aspect of this procedure must be followed in order to restore connectivity to OpsCenter from managed Solaris 11.4 servers newer than 11.4.48.

 

OpsCenter Information Center

One of the most important sections to review is the OpsCenter Information Center, within Oracle's Support Network, for understanding what OpsCenter is and what updates are occurring.

Information Center: Overview of Enterprise Manager Ops Center (Doc ID 1507780.2)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=1507780.2

As of the writing of this article, there have been several CPU (Critical Patch Updates),
stemming from a Log4J vulnerability discovered in the industry in December 2021. 

This is a good place to start, regarding the latest news on OpsCenter.

OpsCenter 12.4 Release

Oracle upgraded OpsCenter to the 12.4 release in 2019. 

Release Announcement - Oracle Enterprise Manager Ops Center 12c Release 4 (12.4.0.0.0) ( Doc ID 2532906.1 ) April 2019
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2532906.1

The documentation is readily available to everyone, without My Oracle Support (MOS):

Oracle Enterprise Manager Ops Center 12c Release 4 (12.4)
https://docs.oracle.com/cd/ops-center-12.4/index.htm

Base Operating System

Ops Center 12c Release 12.4 is supported on Solaris 11.3 as well as Solaris 11.4, making it a well suited management tool which can be used on nearly any piece of Solaris SPARC Hardware.

 

Most installations will be on newer hardware, with security updates available for Solaris 11.4. Solaris 11.4, as of the time of this writing, is on SRU 50... this is the 50'th month after original release of 11.4!

To avoid installing a buggy Solaris 11.4.0 & applying a half-decade of patches to Solaris 11.4.50, Oracle released free for private use Solaris 11.4 CBE or Common Build Environment.

https://blogs.oracle.com/solaris/post/oracle-solaris-114-cbe-downloads-available

 

As of the time of this writing, this starts at Solaris 11.4 CBE starts at SRU 42, but OpsCenter will need to be upgraded to the support repository in order to get the required Perl XML parser.

OpsCenter 12c 12.4 Pre-Requisites

There are OpsCenter bugs which require workarounds, for installation on later releases of Solaris.

There is a known BUG (32548385) with OpsCenter, introduced by Solaris 11.4.30.
The python 'mediator' in Solaris 11.4 SRU 30 is set to 3.7 instead of 2.7. Ops Center requires 2.7.
Ops Center Will Not Start After Upgrading to Solaris 11.4.3- SRU 30 - Svc:/application/scn/ajaxterm:default is Restarting Too Quickly (Doc ID 2760685.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2760685.1

This mediator pre-requisite must be present for OpsCenter to start up on Solaris 11.4.30 and later.

There is a known BUG (33622838) with OpsCenter, introduced by Solaris 11.4.39.
Older release of Perl 5.26 in Solaris 11.4 SRU 30 is removed. OpsCenter EC requires Perl 5.2.2.
Ops Center 12.4 upgrades to Solaris 11.4 SRU 39 on an EC will fail (Doc ID 2826475.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2826475.1

This perl release pre-requisite must be present for OpsCenter to install on Solaris 11.4.39 and later.

OpsCenter 12.4 Installation

OpsCenter should be installed or upgraded to its most recent base version.

A basic installation with a Single Enterprise Controller is readily available:
https://docs.oracle.com/cd/ops-center-12.4/doc.1240/e59965/GUID-0DE73AE5-1B0B-4403-890A-8F632AD30131.htm#OPCSO525

After upgrade or installation, patches should be applied.

[Byzantine Mosaic: Jesus Christ Pantocrator, Courtesy Ricard MN Photography]

OpsCenter 12.4 Critical Patch Updates

Normally, Critical Patch Updates are cumulative, but this ceased to be the case after April 2022.

A circuitous path to follow, to deal with bureaucracy, was un-affectionately referred to as Byzantine.

This is where our Byzantine journey begins!

OpsCenter 12.4 April 2022 Critical Patch Update

The April 2022 CPU resolved a variety of issues, including Log4J.
(The January 2022 release, with Log4J patches, is also bundled in the April 2022 CPU.)

Ops Center 12.4 companion document for the April 2022 CPU (Doc ID 2865470.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2865470.1

Non-intuitively, this refers the user to another document, that says EM-only, but includes OpsCenter:

Critical Patch Update (CPU) Program Apr 2022 Patch Availability Document (EM-only) (Doc ID 2844807.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2844807.1

A "Bundle Patch" was created, which must be downloaded, and applied according to the instructions:

OPSS BUNDLE PATCH 12.2.1.4.210418 Patch 32784652 or later
https://support.oracle.com/epmos/faces/ui/patch/PatchDetail.jspx?patchId=32784652

These are the major bugs which have been resolved:

Bug 33601961 - Ops Center 12.4: CVE-2021-40438 Apache HTTPD server
Bug 33490456 - CVE-2021-2351: UPDATE THE C CLIENT LIBRARY FOR NNE VULNERABILITY
Bug 33735042 - CVE-2021-44832: APACHE LOG4J UPDATE TO 2.3.2, 2.12.4, OR 2.17.1

This must be applied only on an installation of OpsCenter 12.4

OpsCenter 12.4 July 2022 Critical Patch Update

The July 2022 CPU resolved a variety of issues...

Ops Center 12.4 companion document for the July 2022 CPU (Doc ID 2885006.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2885006.1

Non-intuitively, this refers the user to another document, that says EM-only, but includes OpsCenter:

Critical Patch Update (CPU) Program Jul 2022 Patch Availability Document (EM-only) (Doc ID 2867874.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2867874.1&_afrWindowMode=0&_adf.ctrl-state=1b5ay5nont_123#babfaaai

 A "Bundle Patch" was created, which must be downloaded, and applied according to the instructions:

Ops Center UCE patches for Jul CPU 2022 Patch 34332927 or later
https://support.oracle.com/epmos/faces/ui/patch/PatchDetail.jspx?patchId=34332927

These are the major bugs which have been resolved:

Bug 34259326 - Ops Center 12.4: CVE-2022-22720 in Apache 2.4.52
Bug 34259352 - Ops Center 12.4: CVE-2022-22721 in Apache 2.4.52
Bug 34269953 - Ops Center 12.4: Upgrade OpenSSL to 1.1.1o

This must be only applied after the April release.

OpsCenter 12.4 October 2022 Critical Patch Update

The October 2022 CPU resolved a variety of issue...

Ops Center 12.4 companion document for the Oct 2022 CPU (Doc ID 2904332.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2904332.1

Non-intuitively, this refers the user to another document, that says EM-only, but includes OpsCenter:

Critical Patch Update (CPU) Program Oct 2022 Patch Availability Document (EM-only) (Doc ID 2888514.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=298046759019691&id=2888514.1&_adf.ctrl-state=1b5ay5nont_659

A "Bundle Patch" was created, which must be downloaded, and applied according to the instructions:

Ops Center UI/Other patches for Oct CPU 2022 Patch 34611523 or later
https://support.oracle.com/epmos/faces/ui/patch/PatchDetail.jspx?patchId=34611523

These are the major bugs which have been resolved:

Bug 33952830  CVE-2021-23450: DOJO UPDATE TO AT LEAST 1.17.0

This must be only applied after the April and July releases.

OS Upgrade to Solaris 11.4 SRU 48

This may seem counter intuitive, but there is a bug in SRU 48, which disconnects OpsCenter agent from the OpsCenter Proxy Controller, which reflects in a down agent in OpsCenter Enterprise Controller... and before you can fix this bug, the operating system must be upgraded to crash the agent, then the fix can be applied.

Fixing the OpsCenter OS Agent

With application of Oracle Solaris 11.4 SRU 48, the agent fails to connect to the management station.

A good article on the topic is:

Ops Center 12.4: CDOM Agents fail to start after a Solaris upgrade to 11.4 SRU 48 ( Doc ID 2892465.1 )
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2892465.1

At the root cause, there are a variety of bugs identified with SRU48.

Bug 34525568 : OpsCenter 12.4 CDOM Agents fail to start on Solaris SRU11.4.48 due to XMPP
Bug 34560282 : Ops Center Agent won't start after upgrade to 11.4SRU48 with S7 having Global zone
Bug 33876279 Local connections should skip TLS, SASL handshake

The patch to correct this problem, with SRU 48+, can only be acquired from the OpsCenter team, and is not generally available through the Oracle Patch Management system

A Service Request must be filed, specifically asking for the patch:

Patch 34525568

Installing a ISO from ILOM 4.0.3+ using SSH

Abstract:

The SPARC platform have long come with various Lights Out Management (LOM) capabilities, to access the hardware, and provide for access to the OS from underneath, when there is a hardware issue. A more advanced system called Integrated Lights Out Management (ILOM) was later created. With ILOM 4.0.3, a feature was created to allow for the boot from a remove ISO via SSH!

Where to get ISO:

The easiest place to get the most recent version of Solaris, such as the Common Build Edition

https://www.oracle.com/solaris/solaris11/downloads/solaris-downloads.html

Various ISO's are available from Oracle for Solaris:

Where to download Oracle Solaris ISO images and Update Releases (Doc ID 1277964.1) 
https://support.oracle.com/epmos/faces/DocumentDisplay?id=1277964.1

What to do from ILOM:

Copy ISO's to a local directory on a server

a.b.c.d/user$ ls -l /export/home/user/*iso

a.b.c.d/user$ ls -al /u000/P2V/iso/*iso

-rw-r--r--   1 user root   2314731520 May 12  2016 /export/home/user/sol-10-u11-ga-sparc-dvd.iso

-rw-r--r--   1 user root     867020800 May 13  2016 /export/home/user/sol-11_3-text-sparc.iso

-rw-r--r--   1 user root   1018736640 Apr 23  2019 /export/home/user/sol-11_4-text-sparc.iso

-rw-r--r--   1 user root     551464960 Oct 20  2011 /export/home/user/sol-11-1111-text-sparc.iso

Make sure there is connectivity from the ILOM to the Server hosting the ISO

-> set /SP/network/test ping=a.b.c.d
Ping of a.b.c.d succeeded

Set the ILOM Host Storage Device to Remote

-> set /SP/services/kvms/host_storage_device/ mode=remote

Set the username, password, and ISO location

-> cd /SP/services/kvms/host_storage_device/remote

-> set username=user
-> set password=password
-> set server_URI=sshfs://a.b.c.d:/export/home/user/sol-11_4-text-sparc.iso

/SP/services/kvms/host_storage_device=remote
Targets:
Properties:
password = *****
server_URI = sshfs://a.b.c.d:/export/home/user/sol-11_4-text-sparc.iso
username = user

Review Values

-> show /SP/services/kvms/host_storage_device/

/SP/services/kvms/host_storage_device

Targets:
remote

Properties:
mode = remote
status = operational 

Stop Automatic Boot on Host

-> set /HOST/bootmode script="setenv auto-boot? false"

Mount & Boot the Remote ISO

-> start /SP/console -script 

{ok} reset-all
{ok} devalias
...
rcdrom

{ok} ok boot rcdrom

Boot device: /pci@311/pci@1/usb@0/storage@1/disk@0 File and args:
SunOS Release 5.11 Version 11.4.0.15.0 64-bit
Copyright (c) 1983, 2018, Oracle and/or its affiliates. All rights reserved.
Remounting root read/write
Probing for device nodes ...
Preparing image for use
NOTICE: mount: not a UFS magic number (0x0)
NOTICE: mount: not a UFS magic number (0x0)
Done mounting image
USB keyboard
1. Arabic 15. Korean
2. Belgian 16. Latin-American
3. Brazilian 17. Norwegian
4. Canadian-Bilingual 18. Portuguese
5. Canadian-French 19. Russian
6. Danish 20. Spanish
7. Dutch 21. Swedish
8. Dvorak 22. Swiss-French
9. Finnish 23. Swiss-German
10. French 24. Traditional-Chinese
11. German 25. TurkishQ
12. Italian 26. UK-English
13. Japanese-type6 27. US-English
14. Japanese

To select the keyboard layout, enter a number [default 27]:

Additional Information

A good note on this process is available for people with Oracle Support:
How to Install/Re-image a T5-x, S7, T7-x, T8-x, M7-x, or M8-x System Using the sshfs Protocol (Doc ID 2817892.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2817892.1

Fujitsu: SPARC64 Enhancements for 2022 and 2026

[SPARC64 M12 Roadmap, courtesy Fujitsu]

SPARC64 Enhancements for 2022 and 2026

Abstract

Fujitsu produced their first SPARC processors in 1986, setting the stage for decades of continual CPU architectural improvements. With their migration from 32 to 64 bit, they branded their CPU as SPARC64. Fujitsu create the fastest cores on the market in 2017, which still prove extremely competitive today!

Platform

The SPARC64 XII is the basis from small 1U high to mid-range system. More importantly, they can build contiguous & expandable on-premise cloud platforms. The M12 hardware platforms are advanced systems where building blocks can be tied together with high speed interconnects to create a system which can be tolerant of a hardware failure, where OS images can be re-started, even if the OS VM must straddle two different physical chassis blocks.

Roadmap:

Fujitsu published a new roadmap for their SPARC Solaris platforms.

Fujitsu plans on a new SPARC enhancement under Solaris in 2022 time-frame for the SPARC64 M12 platform, with another enhancement planned for 2026! End of sale for their high-end SPARC platforms is projected for 2029 with end of support slated for 2034. The Solaris Operating System roadmap coincides with the Fujitsu M12 platform.

Conclusion

Any platform with a 12 year life expectancy, where no major OS upgrades are expected, offers a degree of stability and value for important mission critical software that is truly unmatched by existing platforms today.