Solaris 11.4: KSplice Arriving

[Solaris Logo, Courtesy Sun Microsystems]

Solaris 11.4: KSplice Arriving!

Abstract:

The concept of patching an OS system has existed since the beginning of computers. At one time, source code needed to be compiled and a system rebooted with the new OS. Pre-compiled patches were later shipped by OS vendors. Patches were often allowed to be applied while the OS in Single User Mode. Sun had created the concept of Live Upgrade, where patching could be done to an alternate boot environment & booted at later convenient time, to reduce downtime to a simple reboot. In 2007, another feature referred to as Deferred Activation Patching allowed for the current running Sun Solaris 10 OS to be patched, while deferring some of the patches until the next boot, leveraging loopback file systems. In 2008, KSplice concept was birthed from MIT for LINUX kernels, to perform some degree of patching, while severely limiting reboots. Oracle purchased Sun and Solaris in 2009. The MIT students won a $100K award, to start up a company in 2009. KSplice was  purchased by Oracle in 2011. The promise of KSplice and rebootless patching for Solaris was understood in concept, but not executed, until recently.

[Oracle Logo, courtesy, Oracle Corporation]

The Road to Solaris:

KSplice team had talked about Solaris support, before the purchase by Oracle, but it was first being integrated into Linux Platinum Support. NetMgt discussed the benefits & differences in KSplice for Linux vs Solaris in 2012. NetMgt reported that Solaris and KSplice Engineers were meeting in 2012. Goals included:

1. Solaris team bringing KSplice technology into OS
2. Reboot-less small fixes via KSplice into Solaris
3. Allow customer to keep patches "up to date" with year long uptime
4. Leverage Synergies with existing philosophies:
   (DTrace allows data path switching without latency or interruption)

The engineering task of merging DTrace & KSplice technologies between Linux & Solaris was seemingly on. In 2015, NetMgt reported discussion of KSplice for Platinum Support Linux with DTrace became a major differentiator for Oracle, but KSplice discussion regarding Solaris was still dark... until 2018.

[Solaris 11.4 Beta image, courtesy Oracle Corporation]

Solaris 11.4 (aka Solaris 12)

Since KSplice was a significant feature, those of us in the industry expected it to arrive in Solaris 12. To casual observers, it became clear that "big bang" approach to OS's was becoming obsolete. Apple started delivering MacOSX, continually, with no major interruptions. Windows started delivering Windows 10 continually, with no major future interruptions planned. Solaris announced they would be entering a similar Continuous Delivery model in 2017. Some reporters spread #FakeNews about Solaris being dead, but those who understood industry trends were watching for features from Solaris 12 to be integrated. The first & second Open Beta release of Solaris 11.4 was done... and it was clear that Solaris 12 features were being bundled into the Solaris 11 stream, but KSplice was conspicuously missing.

KSplice Hints

We started seeing hints of KSplice in 2017. The test cases for spliceadm were now being bundled. Many thanks to Tim Foster, to providing a hint into the future!

KSplice Infrastructure

It seems the infrastructure for KSplice was bundled in Solaris 11.4, but live splices have not been made publicly available. A sample of what we can observe from the manual.

SpliceAdm

The use of the "adm" suffix in commands for administration has become quite well known in Solaris. A new such command magically appeared in the 11.4 - "spliceadm". The manual page says "Splices are fixes for specific bugs that can be applied on a live system" - KSplice arrived! The Interface Stability is declared as "Committed" - it is here to stay!

Freeze and Unfreeze

A new concept was introduced to KSplice in Solaris 10 - freezing. This seems to be a way to allow for automatic updates, limit automatic updates to a particular splice version, or even facilitate rollbacks to a lower numerical splice. This facilitates the downloading of splices to a system, yet restriction of splice activation/rollback until a time of low utilization.

[Former KSplice Team, courtesy MIT News]

The Back Story

A former engineer, Enrico, from Sun/Oracle published the back-story on KSplice for Oracle Solaris. It appears he had worked with Tim [Foster, perhaps?] on the project. He spoke of resolving scaling problems, not experienced under Linux, due to huge stack sizes supported by SPARC Solaris systems while running latency sensitive Oracle Cluster. He also spoke of KSplice & DTrace compatibility - which we discussed earlier in this article with Linux.

Special thanks for those Enrico identified: Jan Setje-Eilers, Scott Michael, Kuriakose Kuruvilla, Pete Dennis, Rod Evans, Ali Bahrami, Mark J Nelson, Xinliang Li,Raja Tummalapalli, Albert White, Adam Paul, and Gabriel Carrillo.

Conclusions:

As an Application Architect in Remote Network & Systems Management, Configuration Management, and Incident Management - I had personally developed & used techniques to deploy code live in arenas where continual uptime was required. It is good to see similar techniques reach into the OS arena.

Sun had previously set a very high bar with Solaris, offering the ability to patch a live system with complete patch application on a reboot, or applying many parts of a system (until eventual reboot.) Some of this flexibility was rolled back with Solaris 11, to see KSplice finally start arriving on the scene with Oracle Solaris 11.4.

While Oracle Solaris is not rolling splices yet, we look forward to 100% uptime with KSplice, in conjunction with using LDom's on SPARC hardware.

The Future - SPARC M8+ & Solaris

[SPARC Logo, Courtesy SPARC International]

The  Future - SPARC M8+ & Solaris

Abstract:

Solaris has been the heart of Sun Microsystems since 1982, with 32 bit SPARC RISC CPU since 1987. Fujitsu joined the SPARC/Solaris community in 1992, with others to follow. In 1995, SPARC went 64 bit, and has been ever since. In 2009, Oracle Corporation purchased Sun and all of it's SPARC & Solaris assets. Oracle & Fujitsu had been releasing processors & Solaris releases in rapid succession, ever since.

[SPARC Roadmap, courtesy Fujitsu]

Fujitsu SPARC Roadmap

In 2017, Fujitsu released one of the fastest processors on the market, which happened to be a SPARC. Since they, they had been very clear about it's roadmap for SPARC & Solaris, reaching out to a new SPARC release in 2020. Their roadmap has not changed in close to a year, so they appear to be on-track.

[SPARC & Solaris Roadmap, courtesy Oracle]

Oracle SPARC Roadmap

Shortly after Fujitsu's SPARC release, Oracle also released their SPARC M8 processor, which became the fastest socket in the world, once again. Also in September, Oracle release a roadmap where there was no future SPARC socket. This has been remedied in Spring 2018, where a snapshot of the official Oracle SPARC roadmap mirrors Fujitsu.

The spread between M6, M7, M8, and M8+ all seem to be spaced about 2.5 years apart, indicating not much of a change in the silicon release schedule from Oracle. Both Fujitsu & Oracle are now both using the same TSC fab for their SPARC silicon, which hints at a degree of consolidation outside of their roadmaps.

As with the way Embedded 10 Gig Ethernet was consolidated on the old T2+ socket, Network Management is wondering if Oracle will finally release the embedded Infiniband on the M8+ sockets. Embedded Infiniband was conspicuously missing from the "Sonoma" S7 release. (This would be a huge bonus to the Engineered Systems, to add a high degree of consolation & increase reliability & increased performance - TODAY.)

More interesting, there is a new hardware category called "Next Generation Storage", linked with the M8+ servers. Oracle had never been a true hardware company before purchasing Sun - they mostly re-sold hardware from third party vendors in their engineered systems, where Oracle felt they could add services value. Seeing SPARC in the roadmap, for engineered systems, is interesting, but will Solaris SPARC assume the role (with Solaris 11.4's new Storage features) or Intel Linux (with new Block volumes in Linux Storage Appliance)?

[Solaris Logo, Courtesy Sun Microsystems]

Oracle Solaris Roadmap

Oracle announced in early 2017 that SPARC & Solaris will move to an Agile Continuous Improvement cycle. Solaris 12 was erased from the roadmap, which was expected with Continuous Delivery, but created an odd amount of uncertainty in the media. Human resources alignments occurred in 2 batches, creating a large amount of uncertainty, but a new M8 processor was released. Assurances that Premier Support would continue to at least 2038!

January 2018, Solaris 10 went on Extended Support, meaning an uplift for patches are required, and 3 more years before patches cease. The final Premier Support patches were released, called 2018-01. The push to Solaris 11 is on!

Solaris 11.3 continued to get monthly SRU's (i.e. Solaris 11.3 SRU 30, Solaris 11.3 SRU 31, etc.) Open Beta of Solaris 11.4 was released, followed by a refresh, and finally an announcement that there will be a major point release of Solaris ever summer moving forward. The latest Solaris 11.4 refresh is nearly all 64 bit with buffer overflow protections, which is quit an accomplishment! The roadmap shows the next release, Solaris Next, which will likely be Solaris 11.5

Conclusions:

As the market uncertainty over SPARC & Solaris continues, new silicon and new operating system releases continue to occur. Long term industry players continue to release hardware. Solaris continues is march forward with aggressive new features, even pushing all components in the operating system to 64 bit. The only promise still outstanding is rebootless patching, possibly with KSplice. Wim, where is Solaris KSplice? You're in charge now, right?

Oracle: Next Generation of Engineered Systems

[Graphic courtesy Oracle Data Center Kickoff]

Oracle's Next Generation Engineered Systems

Abstract:

Larry Ellison: Executive Chairman of the Board and CTO introduces Oracle's 5th Generation of Oracle Engineered Systems. Provide the Highest Performance systems and Lowest Service Price at the core. Oracle effectively targets Cisco UCS, HP, EMC.

Summary of Major Announcements

Oracle Virtual Compute Appliance X5

Converged compute and srorage; Runs all datacenter applications. High Performance and Lowest Purchase Price... Combines compute servers, networking, and storage servers in the same box... highly available and fully redundant Compute Infrastructure: Scalable from 2-25 nodes; Linux, Solaris, and Windows; Network Infrastructure: High speed, low latency, fully configured fabric, integrates to existing Ethernet & Storage Networks Management Infrastructure: Redundant management servers; virtual assembly builder with templates included Half Price Oracle List to Cisco Discount; almost a third price

Oracle Storage Appliance X5

Twice as fast, half as much

1. Extreme Flash Storage Server
2. High Capacity Storage Server

12.8 TB PCIe Flash or 6.4 TB PCIe Flash with 48 TB SAS Disks

Oracle Database Appliance X5

2x Servers: 2x18 cores; 8x32 GB (256GB DIMM); 2x Infninband; 4x 10 Gbit Ethernet Storage: 4x 200 GB Flash for Redo Logs; 4x 400 GB Flash for ODA Accelerators; 16x 4TB Hard Drive (Data + Temp Tables + Archive Logs)

Zero Data Loss Recovery Appliance

Fully automated, point in time recovery, no data loss, thousands of databases Backup and log to another rack, another data center, or to Oracle Public Cloud

Big Data Appliance

Oracle Big Data SQL joins: Hadoop, NoSQL, and Oracle RDBMS

Exalogic Elastic Cloud X5-2

Private Cloud for Applications & Middleware Portability to Oracle Cloud Compute: 2x 18 cores, 256 GB RAM/node, 800 GB Flash/node Network: 40 Gbit InfiniBand internal; 10Gb or 1Gb Ethernet external Storage: 80 TB Disk; 256GB Storgage DRAM

Exadata Database Machine X5

Workloads: Warehousing, OLTP, Database as a Service, In-Memory Database Flash Disks replaced High-Performance Disks because Flash Capacity Increase and Price drop! Elastic Configurations: 2x DB and 3x Storage Servers... Full Rack... Multi-Rack Optimize for: In-Memory Max DRAM; OLTP Equal DB & Flash; Warehouse High Capacity Storage and Compute

Oracle SuperCluster

Two SPARC Options:

1. SuperCluster T5-8
2. SuperCluster M6-32

Same Storage Server and Software as Exadata X5

Data Center of the Future with Public Cloud

Options Include: - Logging Backups to the Cloud - Cloud as Backup Datacenter - Test and Development in Cloud with Production Local - Production in Cloud with Test and Development Local

The Deep Dive Sessions

The following Deep Dive sessions are for both newly announced hardware as well as for some existing software noted at the bottom of this section. Written summaries provided can assist in helping select which videos to watch.

Oracle SuperCluster
Oracle Largest, Most Advanced, and Most Secure Appliance Exadata Storage Grid Firmware based Hypervisor (vs re-purposed Linux OS as Hypervisor) Cloud Tenant Self Service Portal Rule Based Access Control Metering and Limiting by Account for customer's self service  IO Domain Recipes (i.e. Small, Medium, Large selections)  Templates on top of Recipe (Pre-configured Recipe with OS Patches and Application) Extreme Tenant Isolation through Zone, Network Paths, and Disks Automated Compliance Validation of isolation
Oracle Exadata X5-2
Exadata X5-2: Extreme Flash and Elastic Configurations
Oracle Exalogic X5
Oracle Exalogic X5-2 and Exalogic Elastic Cloud Software 12c Engineered system designed to run the mid-tier components Oracle Applications  Java Applications  Fusion Middleware  Exabus Technology, shared with Exadata, which reduced latency between servers. Platform as a Service (Software made available in a cloud) and Infrastructure as a Service deployed on the customer premise.
Virtual Compute Appliance
Oracle Virtual Compute Appliance: Simplify IT and Save Money Goals: Simplify Deployment Reduce Cost Pre-built system which is ready to use in a Data Center with a minimal number of steps Compute Capability: 2 - 25 nodes Software defined network with Dual Redundant InfiniBand Ethernet and FibreChannel external connectivity Active-Passive Management Server ZFS Storage Appliance with Redundant Controllers Self-Service Provisioning of VM's, Storage, and Network Policy Driven Metering and Chargeback RESTful Infrastructure as a Service (IaaS) interface Oracle Enterprise Manager drives IaaS Fault Detection Incident Management Lifecycle Managment Change Managment Search & Compare of VM's Apply Patches Gold Templates Compliance reporting All software is bundled (Linux, Solaris, OEM 12c, Oracle VM, Orchestration, Oracle Virtual Networking, Oracle Trusted Partitioning)
Oracle Database Appliance
Oracle Database Appliance X5-2 Provides everything to deploy a high availability database & application Wizards for simplified deployment Patch Automation (Firmware, OS, Database, Storage, etc.) Oracle High Availability Software Stack (Real Application Cluster or RAC) Affordable with Capacity on Demand Oracle Multitenant Option bundled License In-Memory Database Option bundled License OS and Virtualization Licenses Refreshed hardware, higher consolidation density Oracle Enterprise Manager Plug-In for Monitoring and Management with Analytics across Appliances Same software stack as Exadata for affordable Test and Development
Oracle FS1 Flash Storage System
Oracle FS1 Flash Storage System  Summary of Features 2 - 16 Highly Available Nodes Petabytes of Flash 2M 50/50 Read/Write IOPS 80 GB/sec or 5 TB/minute Data Movement Designed to leverage Flash, not existing Hard Disk solutions. Supports both Flash and Disk, Designed for Flash with Economies of Disk
Oracle Big Data Appliance X5-2
Big Data Appliance Solves problems surrounding: Performance Optimized Hardware Time 30% Quicker to Deploy Cost 21% Less Expensive to Purchase Integration Data Transparently into the Infrastructure Oracle Big Data SQL for simple insertion Oracle Enterprise Manager Compatibility
Oracle for Enterprise Big Data
The Move to Big Data
Oracle Linux
Oracle: A Complete, Independent Linux Vendor Nothing significantly new, basic key points: Oracle Linux Premier Support included with Oracle Hardware Stand-Alone Oracle Linux Premier Support offered for other servers  MyOracleSupport Integrated  Oracle KSplice Bundled (on-line patches, immediately active) Oracle Enterprise Manager included for Patching and Management  Oracle Clusterware Bundled  Oracle Backport Lifetime Sustaining Support (no bug fixes, new hardware support)  Oracle OpenStack bundled  Red Hat Binary Compatibility Delivery on DVD with pure Red Hat or Oracle Unbreakable Linux Kernel. OS Features Oracle Unbreakable Kernel option for newer Oracle Engineered Systems. DTrace Integration from Solaris for Oracle Linux Isolation features: Linux Containers (LXC) similar to Solaris Zones; Docker (for Application) Free to download, use, distribute, update; Pay for production system Oracle VM Templates Differentiation: DTrace and KSplice

Oracle Solaris 11: Session 2: Extreme Engineering - A Technical Update

Online Forum
Oracle Solaris 11:
What's New Since the Launch
April 2012

Abstract:
Oracle released a series of 4 sessions on Oracle Solaris 11: What's New Since the Launch (see Session 1.) After an overview, two Senior Principal Product Engineers for Oracle Solaris discussed a loose description of what Solaris 11 includes and what Solaris 11 Update 1 will include.

• Faster IPS Packaging
• New VM API for Java for faster performance with NUMA (non-uniform memory access) systems
• Better support for FibreChannel and iSCSI in Zones
• Infiniband and Zone integration
• Integration of Zones under Exadata
• IP over InfiniBand for TCP/IP and Zones
• Virtual NIC Migration in Zones

The presentation by  Dan Price and Bart Smaalders follows:

 
Session II - Oracle Solaris 11: Extreme Engineering - A Technical Update
Dan Price and Bart Smaalders, Senior Principal Product Engineers from Oracle Solaris Engineering discuss Extreme Engineering, from a technical perspective.


Install, Patching, Packaging
- automated all major system administration work



Image Packaging System
- Customers would have different patch methodology
- Network software repository
- Cryptographically verified (secure)
- New comprehensive toolset
- Easy to pilot and automate
- Best practices is now default way

Image Packaging System
- fallback to a patch is merely a reboot


Software Lifecycle Management
- Safe Updates, Fast Reboots
- Support Repository Unit
- Once a month heavily tested patch groups
- Patch testing with all Oracle applications & test suites

Image Packaging System
- Integrated Enterprise-Grade Change Management
- Patching to be a lower-skilled job


Boot Environments Powered by ZFS
- no initial investment
- updates as inactive clone (no interruption)
- defer reboot to best time
- trivial to roll back
- fast reboot
- mirror protection during upgrade
- fully integrated with zones
  Solaris 10 zone integration was difficult
  Solaris 11 zone integration was simplified

Sparse zones were killed in Solaris 11
- Sparse Zones produced 2 different implementations
- Various new features were added to Zones and ZFS in order to simplify Zones in Solaris 11


Security in the Cloud
- Defense in depth
- Multi-tenancy design
- Immutable zones
- Encrypted data per tenant
- Can't compromise changed/swapped OS binaries


Built-in Virtualization
- Near-zero overhead
- Delegated Administration  (i.e. boot, reboot, no uninstall)
- Moved from shared stack to exclusive stack (i.e. vnic)
- Network Isolation, control bandwidth in each zone
- No other operating system offers the options of Solaris


Secure Data for Cloud Tenants
- Encrypted on the wire
- Encrypted on the disk
- Individual users data is encrypted with keys, unlocked on login


Oracle Solaris 11: Catching Up
- Meetings about Solaris 12 happening
- Deep integration, testing happening monthly on all Oracle products
- Support Repository Updates (SRU's) tested across all products
- SRU's being delivered on a monthly basis
- Solaris 11 update once a year with new features


Oracle Solaris: Where We're Going
- Solaris 11 Update 1 targeting end of year
- Updates contain new functionality
- Solaris update info in early October
- Oracle World will be a good place to get new info
- SRU's to contain bug fixes


VM 2.0 - Virtual Memory System for the Next Generation
- Enhance virtual memory system
- Scales to hundreds of terabytes
- No user servicable parts required for ZFS
- Update 1 will deliver some new capabilities
- New API's to be delivered in the future
- JVM's may desire additional capabilities
- Memory systems will be ready when new HW is released

IPS and Zones
- IPS is faster than previous packaging
- IPS will increase in speed in the future

Zones enhances on LUN and app data on another LUN
- Trim down number of steps for zones & data
- Automate through zones framework: provision zone on LUN and take care of all details
- Details include: connect to storage, create zfs pool on storage, provision file system, install zone on storage, know what iSCSI address, connecting iSCSI client to server is automatic
- Migrating zone from another host should be easier
- Integrating up/down stack should be easier, support more over time
- Support FiberChannel and iSCSI in Update 1
- The more the automation, the easier for implementation

Infiniband: Zones on Exadata
- Integrate Zones with Infniband
- Patch: RDFP 3 for Zones coming
- Important for Zones in Exadata
- Native Infiniband Performance for engineered systems with Zones
- IP over Infiniband for TCP/IP in Update 1
- IP over Infiniband with Zones in Update 1

Make Zones Parallel Update in IPS
- Update 1 should offer 2x performance of IPS
- Update zones in parallel in Solaris 11, like now done in Solaris 10
- Systems with 10 or 50 zones can be done in time one goes for coffee
- Very short downtime for customers on patching
- Patching happening on cloned boot environments

Integration with with Java
- Major performance improvement over Java 6
- Java 7 is out; Java 8 is in the works
- No tuning required for JVM or OS
- SPECjbb2005 from build 10 to build 138 2.2x improvement
- Optimum cryptography through Java classes for Solaris Cryptographic Framework
- Hints JVM gives to scheduler
- NUMA API's integrated into JVM (vs older SMP sytsems) for 2x performance
- Large Java development in Oracle and Sun - now merged, consolidating features
- Oracle JRocket did an incredible amount of work in Oracle apps
- Oracle JRocket features being merged into Sun HotSpot
- DTrace JRocket probes being merged into Sun HotSpot

Other Notes
- VNIC migrations

KSplice
- Solaris team meeting with KSplice Team
- Solaris team bringing KSplice technology into OS
- Reboot-less small fixes via KSplice into Solaris
- Allows customer to keep patches "up to date" with year long uptime
- Synergies existing philosophy: DTrace allows data path switching without latency or interruption


Summary
- Oracle integrates & test patches with major software, not customer
- Solaris 11, simplify & automate

Ksplice: Kernal Update Without Reboot

[Ksplice image courtesy Linux by Knight]
Ksplice: Kernel Update Without Reboot

Abstract:
Operating Systems normally comprise two distinct layers: the kernel and the user space. Normally, updating the kernel would require a reboot, so the OS can apply a new kernel module. Operating Systems like Solaris created a mechanism called "live update" to update OS Kernel, OS User Space, or even third-party applications (not to mention provide rollback) with merely a reboot. Oracle Solaris 11 facilitates virtually unlimited patch/rollback cycles leveraging ZFS. The new Ksplice tool from Oracle allows for Linux to get closer to Solaris uptime requirements by providing for kernel updates without reboot, leaving OS User Space and Applications to normal reboot or application restart cycles.

Overview:
Ksplice is a feature of Oracle Linux which installs kernel updates on live systems without reboot, it is free with Oracle Premier Linux Support, and is available today. Even IBM an developer network has some nice things to say about Ksplice, owned by Oracle, and provides a detailed description of how it works.

Caveats:
Ksplice will only work on kernel code distributed by Oracle, no third-party open source kernel modules. Ksplice: facilitates kernel updates in the memory of the system; is used in conjunction to Yum or Uptodate for Kernel and OS User Space binaries on-disk; does not provide for a mechanism to update applications.

Example Commands:
Some important commands, highlighted in a recent screen cast from Oracle include:
uptrack-update - allows for kernel version updates on a live system without boot
uptrack-uname - shows updates on the
uptrack-show - shows updates and effective kernel version
uptrack-remove - allows rollback of kernel versions on a live system without boot
/etc/init.d/uptrack - installs Ksplice kernel updates or on-disk kernel on [re]boot; allow on-disk kernel or even allows to automatically install updates dynamically.
http://uptrack.ksplice.com/ - Web GUI interface to see updates on all servers


Screen Cast Video:
This screen cast provides a Ksplice overview, kernel patching demo, as well as a FAQ.


Network Management Implications:
Solaris offers a single reboot for OS Kernel, OS User Space, and Applications updates, while Ksplice provides Oracle Enterprise Linux with a way to increase availability to avoid reboots with OS Kernel updates (while leaving third-party drivers, OS User Space, and Applications with solving the problem by themselves.)

With the ability of modern operating systems like Oracle Solaris and Oracle Linux to provide near 100% availability (with security), network management systems will increasingly leverage these two foundational components, so managed services providers will be able to provide better availability to their customers than ever before.