Oracle Refreshed SPARC T8 Model

Abstract:

In September 2017, Oracle released their flagship M8 RISC CPU, with T8 platforms based upon it! The S5 Core was an amazing accomplishment, 4 instructions per clock cycle, screaming at 5 GHz. Fujitsu simultaneously released their SPARC64 XII CPU, with SPARC M12 Platforms. Solaris 11.3 was supposed to become Solaris 12, but instead was branded as Solaris 11.4 to encourage continuity. The M8+ processors listed on the now unpublished SPARC/Solaris Roadmap, were supposed to be released in 2020. Fujitsu released a roadmap of a SPARC memory increase of 1.5% in 2021, and Fujitsu came out with new memory risers to expand memory by 150%. Oracle matched & passed Fujitsu in 2022/2023 transition.

Recent M8 Platform Additions

The SPARC M8 Platforms, like the T8-2 systems, were given new phenomenal upgrades:

• Double Memory with memory DIMM upgrades from 64 GB to 128 GB 
• Double SAN Storage Throughput from 16 GB to 32 GB HBA Cards
• Quadruple Network Speed Throughput from 25 GBit to 100 GBit NIC Cards
• Live Upgrade for Firmware
• Better Live Migration
  

The expense of the larger DIMMS is dramatic. Let's compare the rough costs.

• 1TB T8-2 Chassis /w 32GB DIMM's can be used as a baseline.
• 2TB T8-2 Chassis /w 64GB DIMM's is similar to buying 2x 1TB T8-2 /w 32GB DIMM's. 
• 4TB T8-2 Chassis /w 128GB DIMM's is similar to buying 4x 1TB T8-2's /w 32GB DIMM's.

The cost of the new increase capacity is not for the faint of heart.

The 2022/2023 T8-2 Component Refresh

Trade rag coverage on SPARC hardware & software updates has been anemic, at best.
How could we see the future, when trade rags were negligent in their reporting?
Oracle released SPARC T8 features between end-2022 to start-2023, forecast in bug notes.

NOV 2022 - Firmware 9.10.5

Oracle worked on 128 GB DIMM's & 6.8TB Flash F640 v4 NVMe PCIe Card (Aura 10 AIC)

•     33752815 NVME/T_IN read bad value 96 with Aura10-SFF Intel in T8 and S7
•     33627980 prtfru displays incorrect DIMM capacity for 128GB DIMMs

MAR 2022 - Firmware 9.10.4.a

Feature Request for Intel NVMe Solid State Drive Assembly Aura10 and improved live migration 

•    33691051 Aura10 SFF support for SPARC
•    33741127 Add a subguest API minor version for migration-class2 bugfix presence

OCT 2021 - Firmware 9.10.3

Improvements to firmware patching, Ethernet Cards, Memory, and improved live migration.

•    33216275 Enable live patching in Hypervisor 1.20.6
•    33171094 Add card identification support for Intel Ethernet Server Adapter I210-T1 OEM Gen to ILOM
•    33005242 Add ereport.hc.dev_info for memory risers
•    32978497 T8-2 YMR VMEM ZL9025 ASCR_CONFIG change to match T8-1 value
•    32929538 Increase maximum chunks to account for larger dimms
•    32925557 Add Oracle ConnectX-6 Dx EN , 2x100GbE, Dual-port QSFP, Crypto and Secure Boot FW ILOM support (Mellanox ConnectX NIC - Mellanox 25G/100GbE Adapter) (NVIDIA ConnectX-6 Dx Network Adapters) (NVIDIA Connect X-6 Dx Datasheet: ConnectX-6 Dx provides up to two ports of 100Gb/s or a single port of 200Gb/s Ethernet connectivity and is powered by 50Gb/s (PAM4) or 25/10 Gb/s (NRZ) SerDes technology.)
•    32920850 HV support for a Tahoe+ migration class
•    32838122 Additional changes needed for live patching of 32625647/31561348 (live firmware patching)

JUN 2021 - Firmware 9.10.2

Feature request for Intel NVMe Solid State Drive 6.4TB Flash Accelerator F640 v3 NVMe PCIe Card (Aura 9 AIC), additional DRAM Support, and Network Cards

• 33691051 Aura10 SFF support for SPARC
• 32074098 Add Aura9 SFF support to SPARC platforms
• 31365759 [ILOM] "hwdiag pci info" does not show Aura8 SFF Subsystem Vendor ID, Subsystem Device ID, Part Number and Description
• 32385998 Aura 9 SFF fail post in some systems
• 32372784 Add timing data for more DRAM types
• 31447119 Device Monitor: Add support for CX-5 OCP,CX-5 2x100G OCP, CX-6 Dx, and CX-6 Dx 50G, and CX-6 Dx 50G Bifurcated cards
• 31365759 [ILOM]"hwdiag pci info" does not show Aura8 SFF Subsystem Vendor ID, Subsystem Device ID, Part Number and Description

APR 2021 - Firmware 9.10.1.a (available in 9.10.1.c)

Feature request for Intel NVMe Solid State Drive 6.4TB Flash Accelerator F640 v3 NVMe PCIe Card (Aura 9 AIC) and Live Firmware Patching

• 31792974 Updating Aura9 Descriptions and adding Samsung v2
• 32182811 Support multiple tcp connections on SSH-DCA port (security scanners)
• 32059898 Enable live patching in Hypervisor 1.20.5

AUG 2020- Firmware 9.10.0.a

Feature request for Intel NVMe Solid State Drive 6.4TB Flash Accelerator F640 v3 NVMe PCIe Card (Aura 9 AIC), Intel NVMe Solid State Drive 6.4TB Flash Accelerator F640 v2 NVMe PCIe Card (Aura 8 AIC), Intel NVMe Solid State Drive 6.4TB Flash Accelerator F640 v1 NVMe PCIe Card (Aura 7 AIC), 100 GbE Support, 128 GB DIMM, and Live Firmware Patching

• 31225789 Bug fix for Intel NVMe Solid State Drive Assembly Aura9
  
• 31388207 After reset /System, mctp_drive failed on Aura8 SFF
• 31225789 Device Monitor: Add support for Intel prototype Aura9
• 31218973 Failed to access Aura9 SSDs via MCTP/SMBus on E2-2C
• 31190919 add aura9/7 aic to i2c test
• 31181415 Enable live patching in Hypervisor 1.20.4
• 31181349 Device Monitor: Add support for Intel/Samsung Aura9
• 30898834 ILOM change request to support Aura9 SFF from Intel/Samsung and AIC from Intel
• 31440160 Add CX-6 Dx 50G iLOM support
• 31404646 Add CX-6 Dx 2x100G iLOM support
• 31388207 After reset /System, mctp_drive failed on Aura8 SFF
• 31388020 Add CX-5 2x100G OCP NIC support to iLOM
• 30793129 libfru DDR4 (SPD_128RW_FORMAT) CRC protection is ineffective
• 30766834 ILOM is not identifying the part number and description of the Cx5 card
• 30738307 DDR4 (SPD_RW128_FORMAT) SPD_Fault_Data Rrecord re-initialized on SP reboot.

 

Conclusion:

Oracle continues to  refresh their SPARC S7 & T8 lines, with Memory & Network capacity, OS Capabilities, and Firmware Capabilities. We can see from the bug trail that it takes A LONG TIME for Oracle's development team to push features to fruition, about 2 years, but updates are still coming. It is great to see one of the fastest platforms on the planet, dating back to 2017, continuing to get hardware and software updates. More is coming.

How Do I Save the LDoms Configuration under Solaris?

 

 Abstract:

Under SPARC Logical Domains, the Hypervisor is actually running in the firmware of the chassis, where the Control Domain sends commands to partition the hardware underneath the OS's. The hypervisor and all settings are completely in memory... which means if there is a power outage, all virtualization configuration can be lost. The ILOM has onboard storage, to hold the LDoms configuration, when saved, and the hypervisor in the firmware is smart enough to request the configuration from the ILOM on boot, and then simultaneously boot all Logical Domains (including the Control Domain.)

List LDom Configurations

To list all Logical Domain Configurations, which were stored to the ILOM:

sun1824-cd/root# ldm list-spconfig
factory-default
@post-migration [current]
default-config
20190301
20191002
20211014
20220908

Note: in the above example, the "@post-migration" means the configuration was saved the last time someone executed a live migration onto or off of this platform, with the "-s" flag for "save config".

Save Logical Domain Configuration

To save a copy of the LDom configuration:

sun1824-cd/root# ldm add-spconfig `date +%Y%m%d`
sun1824-cd/root#

List Saved Logical Domain Configurations

The newly saved logical domain configuration  should show as the Year, Month, Day combination

sun1824-cd/root# ldm list-spconfig
factory-default
@post-migration
default-config
20190301
20191002
20211014
20220908
20230218 [current]
sun1824-cd/root#

 

How do I reset the ILOM from Solaris?

 

How do I reset the ILOM from Solaris?

Abstract:

A Solaris is comprised of an OS to run the applications under, as well as a variety of instrumentation and virtualization. A typical stack includes: ILOM or SP, CDom, LDom, Zone. If you need to reboot the SP (System Processor) or ILOM (Integrated Lights Out Management), it can be done from the base OS which is normally a Control Domain, and is non-intrusive to the applications running on the base OS (as long as the CDom is not trying to save the state of the LDoms or attempting to read-write the configuration.)

Check ILOM UpTime

The ILOM tracks it's uptime and it is available through the IPMI Tool

sun1824-cd/root# ipmitool sunoem cli
Connected. Use ^D to exit.
-> 

-> show /SP/clock

 /SP/clock
    Targets:

    Properties:
        datetime = Fri Feb 17 22:47:14 2023
        timezone = EST (America/New_York)
        uptime = 34 days, 14:45:14
        usentpserver = enabled

    Commands:
        cd
        set
        show

 -> exit
Disconnected

Rebooting the ILOM

The System Processor Card can be rebooted from the base Solaris OS acting as a Control Domain via:

sun1824-cd/root# PATH=/opt/ipmitool/sbin/ipmitool:/usr/sbin
sun1824-cd/root# export PATH
sun1824-cd/root# ipmitool sunoem cli
Connected. Use ^D to exit.
-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP

-> exit
Disconnected

Verify ILOM Reboot

The ILOM reboot can be verified by using the IPMI tool.

sun1824-cd/root# ipmitool sunoem cli
Connected. Use ^D to exit.
-> show /SP/clock

 /SP/clock
    Targets:

    Properties:
        datetime = Sat Feb 18 22:49:25 2023
        timezone = EST (America/New_York)
        uptime = 0 days, 00:07:43
        usentpserver = enabled

    Commands:
        cd
        set
        show

-> exit
Disconnected
sun1824-cd/root#

 

 

NAWK: How can I get uptime & boot time in Solaris?

 

[Solaris Logo, courtesy former Sun Microsystems, Oracle]

Abstract:

This is part of a small article series on lesser known uses of UNIX nawk. Why nawk? This POSIX language has not changed in decades, gets very few code changes, and is mostly bug free in any reasonable operating system. Scripts written in POSIX nawk will basically run, forever. Master this simple language, and there is virtually nothing that can not be done.

What is AWK?

Awk is a scriptable text language, originally written by the authors of UNIX and C: Alfred Aho, Peter Weinberger, and Brian Kernighan. It's concepts are managing Page Header (the BEGIN clause), Page Body (multiple Pattern -> Action clauses), and Page Footer (the END clause.) 

Awk is suitable for building & manipulating any textual or markup text, which is page oriented (i.e. SQL output, Text, Adobe PostScript, Adobe FrameMaker portable maker format, HTML, XML, Microsoft RTF, Microsoft CSV, etc.)

NAWK: How can I get uptime & boot time in Solaris?

The CTime or Epoch Time can be easily retrieved from nawk using the "srand()" function, if there is no seed passed into the function. This number represents the seconds since 00:00:00 on January 1, 1970.

The boot time is the CTime/Epoch Time [in seconds], at the time of the last OS boot, and can be retrieved from "kstat". Simple subtraction and division can provide reasonably accurate conversions.

The following examples show: CTime, Boot Time, Seconds Difference, Minutes Difference, Hours Difference, Days Difference.

Example 1 - Solaris 10:

Solaris10/root# kstat -n system_misc | nawk '
BEGIN { CTime=srand() };
/boot_time/ { BTime=$2; Diff=CTime-BTime };
END { printf "CTime:\t%s\nBTime:\t%s\nScDif:\t%s\nMnDif:\t%s\nHrDif:\t%s\nDyDif:\t%s\n",
      CTime,BTime,Diff,Diff/60,Diff/60/60,Diff/60/60/24 }'

CTime:  1676683570
BTime:  1676106161
ScDif:  577409
MnDif:  9623.48
HrDif:  160.391
DyDif:  6.68297

Example 2 - Solaris 11:

Solaris 11/root# kstat -n system_misc | nawk '
BEGIN { CTime=srand() };
/boot_time/ { BTime=$2; Diff=CTime-BTime };
END { printf "CTime:\t%s\nBTime:\t%s\nScDif:\t%s\nMnDif:\t%s\nHrDif:\t%s\nDyDif:\t%s\n",
      CTime,BTime,Diff,Diff/60,Diff/60/60,Diff/60/60/24 }'

CTime:  1676683533
BTime:  1662691183
SecDif: 13992350
MinDif: 233206
HrDif:  3886.76
DayDif: 161.948

Does the code look similar? It should, Solaris 10 and Solaris 11 are POSIX compliant.

What about GAWK?

GNU's Awk is not 100% compatible with POSIX Nawk, which is a huge disappointment. I have seen code, which worked for decades, never able to run unchanged under GNU's awk.