State of The Art - SPARC S7 & Solaris

[SPARC S7 Processor, Courtesy Oracle Data Sheet]

State of The Art - SPARC S7 & Solaris

Abstract:

The SPARC processor was developed by Sun Microsystems and had existed since the exit of major systems manufacturers from the Motorola 68K environment. Multiple manufacturers had always existed in this environment, to provide to consumers multiple supply chains in this commodity hardware market. The migration from 32 bit to 64 bit computing in SPARC occurred decades ago, as current computing systems still wrestle with the complexities. Oracle ceased producing horizontal scaling CPU sockets once purchasing Sun Microsystems. In June 2016, Oracle decided to re-enter the commodity market with the SPARC S7. NetMgt had published an article, but it was lost, and it was decided it was time to re-publish it again.

[Labeled Die Photo, courtesy Oracle Hot Chips 27 Presentation]

S7 "Sonoma" Floor Plan:

It can be clearly seen that the new die photo shows the use of 2x 4 Core Clusters. The cores are nearly the same 4th generation S4 cores, bundled in it's 32 core sister M7 processor. Glueless Coherency links were bundled, to scale an S7 system from 8 to 16 cores, with little external circuitry. With DDR4 memory interfaces on-chip, latency is cut down by eliminating external chips. Database Analytics Accelerators have been included, although not as many per-core as with the larger M7.

Close to 20% of the space used by an Infiniband network interface. The last time integrating network on-board silicon occurred was with the UltraSPARC T2+ (which integrated 10 Gig Ethernet) - but it was not released to customer facing production system.This was a significant disappointment to NetMgt, since an new Blade system with an Infiniband backplane allowing scalability to thousands of sockets would have been a welcome addition for cloud computing.

[Courtesy, Oracle's SPARC S7 Servers Technical Overview]

Architectural Changes:

The S7 is a smaller processor, returns to 8 cores on a die, similar to the T4 from back in 2011, but even outperforms processors from 2013 with higher cache, clock speed, and 4th generation core. It was designed to be a competitive socket (in price/performance) to commodity proprietary CPU's (instead of being the fastest performing socket in the market.)

[Dual-Die Photo, courtesy Oracle Hot Chips 27 Presentation]

Dual Socket Configuration:

The glueless dual-socket configuration allows for outstanding communication speeds between sockets. It is apparent that more bandwidth may be available over PCIe links than over the un-exposed Infiniband. Without the Infiniband exposed, the S7 looks more like an UltraSPARC IIIi.

[Infiniband Performance, courtesy Oracle Hot Chips 27 Presentation]

Unexposed Infiniband Performance:

The unexposed infiniband offered the possibility of significant packet performance improvement, over a PCIe card, even under high load. Note, the red Sonoma IB line mostly maintaining between 20-60 Millions of Packets per second.

A blade chassis connecting all minimal Sonoma "S7" blades (holding memory & 2 sockets) connecting back to infiniband storage in an MPP environment scaling across thousands of nodes could have been an amazing entry into Cloud Computing or Super Computing environments.

Conclusions:

While  NetMgt welcomes the new low-cost "Sonoma" S7 options, we see the form-factor produced as a "game changer" unrealized, by not placing the chip in a socket, exposing Infiniband, and into a chassis form factor which could most leverage it's strength. Such a socket could have easily replaced & out-performed the Intel based Storage subsystem Oracle's Infiniband native Engineered Systems. A such a socket in a blade chassis would have filled-out a gaping hole in Oracle's systems portfolio. Furthermore, the cost of memory is so high in the chassis that the cost difference between SPARC "Sonoma" S7 and SPARC M7 are marginal if selecting a chassis to perform LDom virtualization. It is a beautiful chip, for what it is, but a great opportunity lost.

ZFS: A Multi-Year Case Study in Moving From Desktop Mirroring (Part 4)

Abstract:
ZFS was created by Sun Microsystems to innovate the storage subsystem of computing systems by simultaneously expanding capacity & security exponentially while collapsing the formerly striated layers of storage (i.e. volume managers, file systems, RAID, etc.) into a single layer in order to deliver capabilities that would normally be very complex to achieve. One such innovation introduced in ZFS was the ability to dynamically add additional disks to an existing filesystem pool, remove the old disks, and dynamically expand the pool for filesystem usage. This paper discusses the upgrade of high capacity yet low cost mirrored external media under ZFS.

Case Study:
A particular Media Design House had formerly used multiple external mirrored storage on desktops as well as racks of archived optical media in order to meet their storage requirements. A pair of (formerly high-end) 400 Gigabyte Firewire drives lost a drive. An additional pair of (formerly high-end) 500 Gigabyte Firewire drives experienced a drive loss within one month later. A media wall of CD's and DVD's was getting cumbersome to retain.

First Upgrade - Migration to Solaris:
A newer version of Solaris 10 was released, which included more recent features. The Media House was pleased to accept Update 8, with the possibility of supporting Level 2 ARC for increased read performance and Intent Logging for increase write performance. A 64 bit PCI card supporting gigabit ethernet was used on the desktop SPARC platform, serving mirrored 1.5 Terabyte "green" disks over "green" gigabit ethernet switches. The Media House determined this configuration performed adequately.

ZIL Performance Testing:
Testing was performed to determine what the benefit was to leveraging a new feature in ZFS called the ZFS Intent Log or ZIL. Testing was done across consumer grade USB SSD's in different configurations. It was determined that any flash could be utilized in the ZIL to gain a performance increase, but an enterprise grade SSD provided the best performance increase, of about 20% with commonly used throughput loads of large file writes going to the mirror. It was determined at that point to hold off on the use of the SSD's, since the performance was adequate enough.

Second Upgrade - Drives Replaced:
One of the USB drives experienced some odd behavior from the time it was purchased, but it was decided the drives behaved well enough under ZFS mirroring. Eventually, the drive started to perform poorly and were logging occasional errors. When the drives were nearly out of capacity, they were upgraded from 1.5 TB mirror to a 2 TB mirror.

Third Upgrade - SPARC Upgraded:
The Ultra60 desktop was being moved to a new location in the media house, a PM (preventative maintenance) was conducted (to remove dust), but the Ultra 60 did not boot in the new location. It was time to move the storage to a newer server.

The old Ultra60 was a nice unit, with 2 Gig of RAM and a dual 450MHz UltraSPARC II CPU's, but did not offer some of the features that modern servers offered. An updated V240 platform was chosen: Dual 1.5GHz UltraSPARC IIIi, 4 Gig of RAM, redundant power supplies, and an upgraded UPS.

Listing the Drives:

After booting the new system, attaching the USB drives, a general "disks" command was run, to force a discovery of the drives. Whether this is needed or not, is not necessarily important, but it is a step seasoned system administrators do.

The listing of the drives is simple to do through

V240/root$ ls -la /dev/rdsk/c*0
lrwxrwxrwx 1 root root 46 Jan  2  2010 /dev/rdsk/c0t0d0s0 -> ../../devices/pci@1e,600000/ide@d/sd@0,0:a,raw
lrwxrwxrwx 1 root root 47 Jan  2  2010 /dev/rdsk/c1t0d0s0 -> ../../devices/pci@1c,600000/scsi@2/sd@0,0:a,raw
lrwxrwxrwx 1 root root 47 Jan  2  2010 /dev/rdsk/c1t1d0s0 -> ../../devices/pci@1c,600000/scsi@2/sd@1,0:a,raw
lrwxrwxrwx 1 root root 47 Mar 25  2010 /dev/rdsk/c1t2d0s0 -> ../../devices/pci@1c,600000/scsi@2/sd@2,0:a,raw
lrwxrwxrwx 1 root root 47 Sep  4  2010 /dev/rdsk/c1t3d0s0 -> ../../devices/pci@1c,600000/scsi@2/sd@3,0:a,raw
lrwxrwxrwx 1 root root 59 Aug 14 21:20 /dev/rdsk/c3t0d0 -> ../../devices/pci@1e,600000/usb@a/storage@2/disk@0,0:wd,raw
lrwxrwxrwx 1 root root 58 Aug 14 21:20 /dev/rdsk/c3t0d0s0 -> ../../devices/pci@1e,600000/usb@a/storage@2/disk@0,0:a,raw
lrwxrwxrwx 1 root root 59 Aug 14 21:20 /dev/rdsk/c4t0d0 -> ../../devices/pci@1e,600000/usb@a/storage@1/disk@0,0:wd,raw
lrwxrwxrwx 1 root root 58 Aug 14 21:20 /dev/rdsk/c4t0d0s0 -> ../../devices/pci@1e,600000/usb@a/storage@1/disk@0,0:a,raw

The USB storage was recognized. ZFS may not recognize the drives, when plugged into different USB ports on the new machine. ZFS will see the drives through the "zpool import" command.

V240/root$ zpool status
no pools available

V240/root$ zpool list
no pools available

V240/root$ zpool import
  pool: zpool2
    id: 10599167846544478303
 state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://www.sun.com/msg/ZFS-8000-EY
config:

        zpool2      ONLINE
          mirror    ONLINE
            c3t0d0  ONLINE
            c4t0d0  ONLINE

Importing Drives on New Platform:
Since the drives were taken from another platform, ZFS tried to warn the administrator, but the admin is all to well aware that the old Ultra60 is dysfunctional and the importing the drive mirror is exactly what is desired to be done.

V240/root$ time zpool import zpool2
cannot import 'zpool2': pool may be in use from other system, it was last accessed by Ultra60 (hostid: 0x80c6e89a) on Mon Aug 13 20:10:14 2012
use '-f' to import anyway

real    0m6.48s
user    0m0.01s
sys     0m0.05s

The drives are ready for import, use the force flag, and the storage is available.

V240/root$ time zpool import -f zpool2

real    0m23.64s
user    0m0.02s
sys     0m0.08s

The pool was imported quickly.

240/root$ zpool status
  pool: zpool2
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on older software versions.
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zpool2      ONLINE       0     0     0
          mirror    ONLINE       0     0     0
            c3t0d0  ONLINE       0     0     0
            c4t0d0  ONLINE       0     0     0

errors: No known data errors

V240/root$ zpool list
NAME     SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
zpool2  1.81T  1.34T   480G    74%  ONLINE  -

The storage movement went very well to the existing SPARC server.

 

Conclusions:
ZFS for this ongoing engagement has proved very reliable. The ability to reduce rebuild time from days to seconds, upgrade underlying OS releases, retain compatibility with older file system releases, increase write throughput by adding consumer or commercial grade flash storage, recover from drive failures, and recover from chassis failure demonstrates the robustness of ZFS as the basis for a storage system.