Spectre - SPARC Solaris: The Safe Choice

Spectre - SPARC Solaris: The Safe Choice

Abstract:

As the industry continues to struggle with Meltdown, a second vulnerability family appeared referred to as Spectre. As of this article publication, there are 4 variants of Spectre, the latter two variants referred to as Spectre-NG. All SPARC systems are safe, if the most recent systems are on the most current firmware & OS releases. As of this publishing, the latest application/OS & firmware patches fixes the first two. The later 2 does not affect SPARC, as the rest of the Intel and other CPU communities are struggling with their cloud and local server infrastructures.
 

[Spectre logo, courtesy solaris.wtf]

Spectre 

Spectre comes in 4 variants, the first 2 and next 2 identified as of the publishing of this article.

Spectre v1

Upgrade firefox to 57.0.4 or greater for protection (i.e. bundled in recent Solaris 11.3 updates.

Unpatched super-scalar CPU's (i.e. SPARC T4, T5, M6, M7, S7, M8, M10, M12) could possibly be exploited by CVE-2017-5753.

Spectre  v2

 A quick summary on Stack Exchange on how Spectre works:

the attacker tricks the speculative execution to predictively execute instructions erroneously. In a nutshell, the predictor is coerced to predict a specific branch result (if -> true), that results in asking for an out-of-bound memory access that the victim process would not normally have requested, resulting in incorrect speculative execution. Then by the side-channel, retrieves the value of this memory. In this way, memory belonging to the victim process is leaked to the malicious process.

Unpatched super-scalar CPU's (i.e. SPARC T4, T5, M6, M7, S7, M8, M10, M12) may be exploited by CVE-2017-5715.

Spectre v3a

All 64 bit SPARC is immune to CVE-2018-3640 .

Spectre v4

All 64 bit SPARC is immune to CVE-2018-3639 .

[SPARC Logo, courtesy SPARC International]

SPARC

Modern 64 bit SPARC variants come in 2 classes: Scalar and Super-Scalar

[Sun Microsystems Logo, courtesy Sun Microsystems]

Sun UltraSPARC

Older Sun UltraSPARC 64 bit Servers do not have the CPU feature which could possibly be exploited and were not vulnerable... they did not issue speculative instructions. Oracle had purchased Sun, so their support channel can provide a definitive explanation. Performance was mostly driven on these servers leveraging SMP chassis, Multi-Core sockets, and large memory footprints.

[Oracle Logo, courtesy Oracle Corporation]

Oracle SPARC

Newer Oracle SPARC Solaris servers are possibly vulnerable, if you are running a modern CPU which initiates speculative instructions (i.e. T4 or newer) while older 64 bit CPU's are not vulnerable. It has been reported on Solaris WTF that "Spectre (CVE-2017-5753 and CVE-2017-5715)" has been fixed in firmware (i.e. T4: 8.9.10 or greater; T5, M5, M6: 9.6.22a or greater; M7, S7, M8: 9.8.5c or greater.)

The short story, a firmware patch for CPU's newer than T4 are required and the impact is very minor in performance, according to the previous blog. Stock Firefox as shipped with Solaris 10 is vulnerable to Spectre v1, Solaris 11 fixed Firefox vulnerability early 2018, so users should migrate to Solaris 11.

[Fujitsu Logo, courtesy Fujitsu corporation]

Fujitsu SPARC

Sun and Oracle are not the only 2 vendors, who have produced 64 bit SPARC platforms. Newer Fujitsu SPARC Servers are also super-scalar, possibly vulnerable to Spectre v2 (CVE-2017-5753), and have been been fixed in firmware (i.e. M10: XCP2351; M12: XCP3051.)

Conclusions:

If you are using an older Sun UltraSPARC server, you are OK. If you are running a newer Oracle SPARC (i.e. T4 or newer) server, you should update Firefox on Solaris 10 or get on the latest Solaris 11 release to be protected from Spectre v1. For the same class of hardware, apply firmware patches available today to protect from Spectre v2. SPARC is immune to Spectre v3 & v4. Get with your Oracle support for the first 2 variants (doc id 2349278.1) and second 2 variants.

State of The Art - SPARC S7 & Solaris

[SPARC S7 Processor, Courtesy Oracle Data Sheet]

State of The Art - SPARC S7 & Solaris

Abstract:

The SPARC processor was developed by Sun Microsystems and had existed since the exit of major systems manufacturers from the Motorola 68K environment. Multiple manufacturers had always existed in this environment, to provide to consumers multiple supply chains in this commodity hardware market. The migration from 32 bit to 64 bit computing in SPARC occurred decades ago, as current computing systems still wrestle with the complexities. Oracle ceased producing horizontal scaling CPU sockets once purchasing Sun Microsystems. In June 2016, Oracle decided to re-enter the commodity market with the SPARC S7. NetMgt had published an article, but it was lost, and it was decided it was time to re-publish it again.

[Labeled Die Photo, courtesy Oracle Hot Chips 27 Presentation]

S7 "Sonoma" Floor Plan:

It can be clearly seen that the new die photo shows the use of 2x 4 Core Clusters. The cores are nearly the same 4th generation S4 cores, bundled in it's 32 core sister M7 processor. Glueless Coherency links were bundled, to scale an S7 system from 8 to 16 cores, with little external circuitry. With DDR4 memory interfaces on-chip, latency is cut down by eliminating external chips. Database Analytics Accelerators have been included, although not as many per-core as with the larger M7.

Close to 20% of the space used by an Infiniband network interface. The last time integrating network on-board silicon occurred was with the UltraSPARC T2+ (which integrated 10 Gig Ethernet) - but it was not released to customer facing production system.This was a significant disappointment to NetMgt, since an new Blade system with an Infiniband backplane allowing scalability to thousands of sockets would have been a welcome addition for cloud computing.

[Courtesy, Oracle's SPARC S7 Servers Technical Overview]

Architectural Changes:

The S7 is a smaller processor, returns to 8 cores on a die, similar to the T4 from back in 2011, but even outperforms processors from 2013 with higher cache, clock speed, and 4th generation core. It was designed to be a competitive socket (in price/performance) to commodity proprietary CPU's (instead of being the fastest performing socket in the market.)

[Dual-Die Photo, courtesy Oracle Hot Chips 27 Presentation]

Dual Socket Configuration:

The glueless dual-socket configuration allows for outstanding communication speeds between sockets. It is apparent that more bandwidth may be available over PCIe links than over the un-exposed Infiniband. Without the Infiniband exposed, the S7 looks more like an UltraSPARC IIIi.

[Infiniband Performance, courtesy Oracle Hot Chips 27 Presentation]

Unexposed Infiniband Performance:

The unexposed infiniband offered the possibility of significant packet performance improvement, over a PCIe card, even under high load. Note, the red Sonoma IB line mostly maintaining between 20-60 Millions of Packets per second.

A blade chassis connecting all minimal Sonoma "S7" blades (holding memory & 2 sockets) connecting back to infiniband storage in an MPP environment scaling across thousands of nodes could have been an amazing entry into Cloud Computing or Super Computing environments.

Conclusions:

While  NetMgt welcomes the new low-cost "Sonoma" S7 options, we see the form-factor produced as a "game changer" unrealized, by not placing the chip in a socket, exposing Infiniband, and into a chassis form factor which could most leverage it's strength. Such a socket could have easily replaced & out-performed the Intel based Storage subsystem Oracle's Infiniband native Engineered Systems. A such a socket in a blade chassis would have filled-out a gaping hole in Oracle's systems portfolio. Furthermore, the cost of memory is so high in the chassis that the cost difference between SPARC "Sonoma" S7 and SPARC M7 are marginal if selecting a chassis to perform LDom virtualization. It is a beautiful chip, for what it is, but a great opportunity lost.

Hot Chips 24: SPARC T5 Overview

[SPARC T5 Processor, courtesy Oracle 2013-03-26 Announcement]

Abstract:
Every year, the best of engineering talent comes together in academia for Hot Chips conference, to present the best system designs. During Hot Chips 24, Session 9 - the SPARC T5 was presented by Sebastian Turullols and Ram Sivaramakrishnan from Oracle on Wednesday, August 29, 2012. This processor was released 6 months later, by Oracle with their T5 systems on Tuesday March 26, 2013.

Video Presentation:
The video presentation of Session 9 was conducted by Fujitsu, Oracle, and IBM. The middle portion of the presentation, starting at 30 minutes, includes the Oracle presenters.


Slide Presentation:

The following screen shots were taken of the presentation. The full presentation is available here.

[4x memory controllers are capable of a peak of 128 GigaBytes per Second]

[Acceleration of Contended Locks; linked list of address requests; all requests satisfied atomically]

[Directory based level 3 indices, tracked up to 8 sockets on an on-chip SRAM]

[7 links with 14 lanes per link between sockets]

[C2C is a sharing cache]

[28 GigaBytes bandwidth between nodes; allows for aggregation of throughput via intermediate node]

[OLTP workload is extremely shared, makes workloads very difficult to scale]

[Elastic Mode includes all of the power saving features automatically]

[Solaris makes frequency request from hypervisor; cores reduced or cycle skipping used]

[Frequency and Voltage pushes performance maximum possible by the system]

Question-Answer Session: The following information was provided during the Question-Answer session, from the audience.

• Voltage required for 3.6GHz varies from part to part. 0.95-1v is needed.
• There is one voltage supply for all cores; one common PLL for the entire chip. Cycle skipping is used to vary cycle rate.
• Low latency clustering port leverages "Allocated DMA Feature"
• T5 no longer has an integrated 10 Gigabie Ethernet Controller.

Tab Update: Solaris - New T5 and M5 Platforms!

With the new Oracle T5 and Oracle M5 processors released from Oracle yesterday evening, two new white papers have hit Network Management blog, covering the architecture for solution integrations.

Solaris Reference Material
...
2013-03 [PDF] Oracle's SPARC T5-2, T5-4, T5-8, and T5-1B Server Architecture
2013-03 [PDF] Oracle's SPARC M5-32 Server Architecture

SPARC T5 Partner Cast

[SPARC T5, courtesy Oracle Corporation]

SPARC T5 Partner Cast
Abstract:
John Shell, Senior Director of Systems Partner Enablement for Oracle, hosts Rick Hetherington, VP of Hardware Development for Oracle, to discuss Oracle’s SPARC T5 Processor.

Interview Notes:
SPARC T4 Summary:
- S3 Cores, clocked at 3.0 GHz
- Crypto Cores inserted into Core with additional instructions, highest level of crypto performance in the industry

SPARC T5 Summary:
- Same S3 Cores
- Faster clock rate, more cores
- Improvement in 1 year

Solaris Summary:
- Solaris 10 and Solaris 11 have both been optimized for T4 & T5
- Enhanced security library, thread scheduling, improved vm system
- Adding additional features into SPARC for Oracle stack optimizations
- Industry leading performance on long list of ISV's

Partner Summary:
- Oracle continues to invest hevily into SPARC and Solaris
- Continual flow of ever improving SPARC processors
- SPARC processors will continue to form heart of engineered systems

Q-A
- Is T5 a Systems on a Chip? Yes
- What is the value of 28nm? Transistors are smaller, more functionality on die, improved cycle time
Media:

SPARC T5 Deep Dive

[SPARC T5 Microdie, courtesy Oracle Corporation]

Abstract:
The SPARC platform has existed for over 25 years, with continued binary compatibility. The latest of these processors, the SPARC T5, is about to be released. Oracle published a short written interview with the Rick Hetherington, who is leading the charge on newer SPARC processors.

SPARC T5 Deep Dive:
Oracle regularly publishes short question-answer articles, called Deep Dives, regarding new platforms during the release phase. This latest SPARC T5 release is no different. An overview and questions are listed below. This is the article, for the answer to the questions.

Rick Hetherington, Oracle’s vice president of hardware development, manages a team of architects and performance analysts who design Oracle’s M- and T-series processors. In this interview, Hetherington describes the technical details of the new SPARC T5 processor and expands on the process that is used to design these innovative chips.

Q: What were the design objectives of the SPARC T5 processor?
...
Q: So what’s new in the SPARC T5?
...
Q: This is mainly a performance increase story then?
...
Q: Does the SPARC T5 also support both single-threaded and multi-threaded applications?
...
Q: Was there anything in the design process that surprised you, or did things go the way you expected?
...
Q: And how do you know which workloads you want to model on?
...
Q: What do you mean when you say trace?
...
Q: What do you consider innovative in the SPARC T5?
...
Q: What kinds of applications will benefit the most from the SPARC T5?
...
Q: What about the security features in the T5?
...
Q: I’d like to know if there was any kind of optimization with Solaris 11.
...
Q: And the Solaris binary compatibility still applies?
...
Q. What's the most important thing you want customers to know about the SPARC T5 processor?

...

Conclusions:
The SPARC platform continues to seesignificant archtiectural improvements - the T5 is no exception, with combined higher clock speed and double the processing cores. This is tremendously good news to the telecommunications industry, which has been waiting for an upgraded Open Systems platform.

SPARC T5: Architect Rick Hetherington

[SPARC T5 image, courtesy Oracle Corporation]

SPARC T5: Architect Rick Hetherington

Rich Hetherington discusses in video and and deep-dive interview regarding the new SPARC T5 processor from Oracle.

Some of the question highlights:

• Q: What were the design objectives of the SPARC T5 processor?
• Q: So what’s new in the SPARC T5?
• Q: This is mainly a performance increase story then?
• Q: Does the SPARC T5 also support both single-threaded and multi-threaded applications?
• Q: Was there anything in the design process that surprised you, or did things go the way you expected?
• Q: And how do you know which workloads you want to model on?
• Q: What do you mean when you say trace?
• Q: What do you consider innovative in the SPARC T5?
• Q: What kinds of applications will benefit the most from the SPARC T5?
• Q: What about the security features in the T5?
• Q: I’d like to know if there was any kind of optimization with Solaris 11.
• Q: And the Solaris binary compatibility still applies?
• Q. What's the most important thing you want customers to know about the SPARC T5 processor?

Vendors, Systems, and Processors Update

[HotChips 2012 agenda exerpt, courtesy HotChips 24]

Vendors, Systems, and Processors Update

Normally, we don't release a consolidated update on the industry more than once a month, but there has been some significant updates.

Hot Chips 24: A Symposium on High Performance Chips is right around the corner, and the agenda looks pretty exciting1

IBM: POWER up?
By the time HotChips 24 arrives, POWER 7+ should be about 1 year late, as Fujitsu SPARC remains #1 for over a year in the HPC charts.

Is IBM really going to talk-up POWER 7+, one year late, without releasing it? It is looking a lot like what happened to Sun Microsystems with their ROCK processor, which was killed not long after there were multiple presentations on it, around the time of Oracle acquisition of Sun Microsystems.

IBM will also talk about their zNext processor, whatever that might be. Will POWER 7+ ever see the light of day?

Oracle: The SPARC is HotAround the time the industry was expecting IBM POWER 7+, Oracle release the SPARC T4 processor.

About 1 year later, during the same time that IBM will be talking about POWER 7+, Oracle is projected to release their SPARC T5 processor. The industry is hoping that Oracle will fulfull it's projection to release SPARC T5 in 2012, about 6 months ahead of time on their roadmap.

The SPARC T5 is supposed to be a glue-less 8 socket processor, adhering to SPARC V9 open standard, certified by SPARC International. Different extensions are projected to be included, such as Oracle RDBMS number calculations in hardware and compression engines... both which will dramatically increase the performance of Oracle RDBMS's.

With the increase of Oracle RDBMS's also comes the dramatic increase in performance of software with embedded databases (which is basically everything enterprise grade.) Oracle has determined to sit on the top of the Enterprise Software performance stack and SPARC seems to be the delivery mechanism.

Why is Open Standards important in platforms? When a single vendor comes under pressure and can't deliver (i.e. IBM POWER 7+) - other vendors are free to "pick up the slack", earn a little money, and produce something of additional value for the consumer.

Fujitsu: SPARC On Top Today, Intending to Stay On Top
Fujitsu has a long history of producing SPARC CPU's, both for Sun Microsystems as well as for themselves. Fujitsu manufactured the first Sun SPARC processor, manufactured high-end systems for Sun and Oracle for the past half-decade, and has been holding the #1 performance spot on the HPC 500 list.

Fujitsu released several iterations of their own SPARC CPU for massive super-computer (SPARC64 VIII fx, SPARC64 IX fx) Linux systems, as well as processors high-end (SPARC64 V, VI, and VII) Solaris systems. During HotChips 24 - they are projected to talk about their SPARC64 X processor!

The industry is hoping for a Solaris variant, based upon OpenSolaris fork like Illumos, to unify the Fujitsu and Oracle platforms, but there are no rumblings about that.

[ARM TrustZone technology, courtesy ARS Technica]

AMD: Embedding ARM in x64?
After reading about the Dell inclusion of ARM as an enterprise blade platform, the only thing more shocking would be the inclusion of ARM in a mainstream CPU vendor. Well, that day has come: ARM is coming to AMD Opteron.

The use ARM in the AMD world seems to be targeting virtual computing. The TrustZone feature of ARM may prove interesting for booting hypervisors or providing DRM (digital rights management).

[Fujitsu PrimeHPC node, courtesy The Register]

HPC: Battle of the RISC's
Intel and AMD systems long ago took the top HPC spots. There was a general movement towards using graphics card co-processors to boost scores with specialized software. Some thought that the inclusion of ARM would help for future HPC systems, but with Fujitsu SPARC sitting on the top for a year, without any special co-processors, one may wonder whether graphics card vendors and special co-processor vendors have decided to sit out the super computer market, for awhile, since Fujitsu keeps upping the performance of their long-living SPARC open architecture.

Network Management Connection
With the rise of SPARC and ARM, one may wonder the impact for Network Management. ARM seemingly sits on most mobile devices, which all need to be managed. SPARC seemingly sits on the fastest Enterprise and HPC Systems. Network Management tool vendors will need to leverage these capabilities or at least manage them. Proprietary Intel is the volume proposition. AMD is the second-sourcing proposition for the proprietary Intel platform.

No network management vendor ignoring Intel, AMD, ARM, or SPARC are worth their weight in printed code.