SPARC: Life in the Fast Lane - 10 Months Later

[aggregate computing power in the HPC list, by processor architecture]
SPARC: Life in the Fast Lane - 10 Months Later

Abstract:

SPARC, being a standard by which any vendor can create a binary compatible processor to leverage readily available applications, has been at the top of the HPC charts in the past, but noton the top for some time. About 10 months ago, Japan skyrocketed to the top of the Top 500 High Performance Computing systems world-wide. Fujitsu, a multi-vendor and open systems equipment manufacturer, designed a new SPARC processor. Fujitsu designed an Interconnect, CPU, Hardware Platform, File System, and Overall MPP System to make their mark.



[Rikagaku Kenkyusho (Riken) research lab in Kobe, Japan - image from The Register]

The Fujitsu MPP System

The Japanese Government funded Fujitsu to implementation the K machine under Project Keisoku. The Japanese government often referred to it as the Next Generation Supercomputer Project. The massively parallel processing (MPP) system cost approximately $1.2bn and was implemented using standard general purpose CPU's, without the need to leverage graphic co-processor cards. The Register writes in November of 2011.

The K super is based on the "Venus" Sparc64-VIIIfx processor designed by Fujitsu and fabbed by Taiwan Semiconductor Manufacturing Corp. The eight-core Venus chip clocks at 2GHz and delivers 128 gigaflops per chip, has a thermal efficiency of around 2.2 gigaflops per watt, and dissipates around 58 watts.

The K super has 22,032 four-socket blade servers fitted into 864 server racks to bring 705,024 cores...

Many newer systems are using graphical co-processor cards, in order to meet higher floating point processing requirements, so this system was a real shocker to the rest of the community. The former #1 contender established themselves about 8 months earlier, the Chinese Government funded Tianhe-1A - a CPU-GPU hybrid system using Intel CPU, NVidia GPU, and Chinese designed SPARC CPU's.

Many speculated that HPC systems could no longer compete without GPU's, yet Fujitsu's SPARC platform proved them wrong, by a long run. The Fujitsu general purpose SPARC super computer has remained #1 for longer than the former CPU-GPU hybrid.


[Fujitsu's PrimeHPC FX10 upgrade to the K super computer - image from The Register]
The Building Blocks

Fujitsu, who has a long history of building mainframe class systems, as well as SPARC based processors, decided to commoditize their #1 K super computer, in the form of the PrimeHPC FX10. The Register also writes about it's massive scalability in November of 2011:

The PrimeHPC FX10 machine will scale from 4 to 1,024 cabinets, sporting between 384 and 98,304 nodes. In the K architecture, each socket on the four-socket blade is a unique node in the cluster. This is also true for the FX10 super.

The new building block uses 16 core processors, instead of 8 core processors, granting the overall commercial system a total high-end capacity of 1,572,864 cores - double the capacity of their #1 super computer, just released months earlier.

[Top 500 super interconnects share, by system capacity - courtesy The Register]
The Interconnect

The Interconnect is sometimes the least interesting component in a system. Some systems use 1 or 10 gigabit ethernet, others use infiniband. In the case of the Fujitsu system, their "secret sause" is their proprietary "Tofu" interconnect.

[Fujitsu's FEFS is based upon Oracle Lustre]

The File System

The File System used by the Fujitsu's Super Computers are both based upon Sun/Oracle Lustre file system - called FEFS or Fujitsu Exabyte File System.


[The PrimeHPC blade server with Tofu interconnect chips on the left - courtesy The Register]
Four Nodes per Blade Card

In the MPP world, a node is sometimes considered a chassis, other times a card. In Fujitsu's SPARC64 PrimeHPC system, a node is built around a socket on a card. With the K super computer, there were 8 cores per socket. With the new Prime HPC blade, each socket holds 16 cores.


[The SPARC64 IXfx floorplan - courtesy Fujitsu]
The SPARC Socket

The SPARC VIIIfx and IXfx are not terribly unique in the marketplace. Sun and Oracle had offered 8 and 16 core processors before Fujitsu - but Sun, Oracle, and Fujitsu were the market leaders - and all three companies were/are doing an Open Systems based SPARC design. While the rest of the HPC market seemed to concentrate on proprietary CPU vendors (perhaps this made these three companies unique.

SPARC Comparison	SPARC64TM VIIIfx	SPARC64TM IXfx
Number of cores	8 cores/socket	16 cores/socket
Clock frequency	2 GHz	1.848 GHz
L1 Cache	I: 32KB/core, D: 32KB/core	I: 32KB/core, D: 32KB/core
L2 Cache	6 MB (Shared cache)	12 MB (Shared cache)
Peak performance	128 Gigaflops	236.5 Gigaflops
Memory throughput	64 GB/s	85 GB/s
Power consumption	58 Watts	110 Watts
Process	45 nm	40 nm
Die size	22.7 mm × 22.6 mm	21.9 mm × 22.9 mm
Number of Transistors	~ 760 million	~ 1.87 billion

[The SPARC64 IXfx core - courtesy Fujitsu]
The SPARC Core

The SPARC64 core is an extension of the SPARC V9 architecture. Various enhancements were made by Fujitsu to each core. Some of the features are as follows:

• 64 Bit Processing


• Standard SPARC-V9 instruction set, with enhancements


• Floating Point Registers expanded from 32 to 256


• Combined Integer and Floating Point Unit


• 2 SIMD / Cycle (Single instruction, multiple data)


• 8 Floating Point / Cycle


• New Floating Point Trigonometric functions


• New Floating Point Reciprocal Approximation of divide/Square-root


• New Floating Point Minimum and Maximum Operations

[The SPARC64 history and roadmap - courtesy Fujitsu]
The SPARC Roadmap

The SPARC64 processor has a long history, which pre-dates the acquisition by Fujitsu. The latest processor appears to be just another in the long progression of the SPARC64 line. It appears that Sun dropped SPARC out of the HPC arena, after being purchased by Oracle - but it also appears that Fujitsu decided to aggressively pursue this arena.

Conclusions

Both Oracle and Fujitsu are independantly pursuing SPARC in disjoint, non-overlapping, markets. They are not the only vendors creating new production quality SPARC processors (as noted by the former #1 HPC system from China.) SPARC appears to have a long road ahead, being implemented by multiple vendors, and each implementation performing best in it's class.