Joyent: Encapsulating Linux through Docker into a Zone

[Solaris 11 Launch image, courtesy Oracle]

Abstract:

Virtualization has been available in the UNIX OS world. The creation of users in a time sharing environment, to isolate executable threads from one another as well as protect files in an underlying file system started the journey. The creation of the Virtual File System, where disks could me mounted anywhere in a file system tree (instead of drive letter) revolutionized computing to allow those systems to grow in the shared environment! The creation of "chroot" so an application could run in it's own file system space, made an application "feel" like it is on a dedicated system. The merging of SVR4 into Solaris created a robust multi-processor infrastructure to host multi-user and  multi-tenant systems. The creation of Zones under SVR4 Solaris 10, further extrapolated the original concepts of the UNIX "chroot", isolating CPU, Memory, Users, Storage - effectively making a single instance of the Solaris OS truly multi-tenant. The creation of Branded Zones for Linux and Solaris came later, offering entire operating systems to be encapsulated under Intel and SPARC Solaris systems. Newer proprietary technologies continue to enter the horizon.

[Oracle Linux, courtesy Oracle]

The Linux Problem

People participating in the Linux ecosystem are interested in creating new raw environments,  isolated to their operating system under proprietary Intel processors, to supply a reasonable replacement for mature infrastructure. These replacements constitute very long efforts, which often never really get completed. Veterans understand the benefit of good engineering and can often take systems "to the next level." Vendors like Oracle had taken Linux, ran their applications on top of it, and supplied the patches necessary to keep Linux stable.

Joyent: Zones(KVM and Linux)

Former employees of Sun Microsystems continue to do the heavy lifting in the industry. Network Management wrote about Joyent's efforts to port KVM into Solaris Zones under their SmartOS, based upon Illumos. Illumos originated from Sun Microsystem's OpenSolaris project (which became the basis of Oracle's Solaris 11.)

[Solaris Zone/Container concept, courtesy former Sun Microsystems]

Joyent: Zones(Docker and Linux)

One might expect that Cloud companies who are obsessed with Virtualization like Joyent would continue their quest for a "better cloud". In 2015, Joyent released a presentation on the porting of Docker to encapsulate Linux into a Zone... using the same SmartOS based upon Illumos, which found it's roots in Sun Microsystem's OpenSolaris.

Docker and the Future of Containers in Production from bcantrill

For Joyent, The Cloud means chasing every container technology and integrating it into SmartOS, to give their customers choice, while simultaneously utilizing their infrastructure as efficiently as possible.

Conclusion

SVR4 UNIX and Sun Solaris developers have a long history of virtualization. The success story of Joyent in "Cloud" environments continues to lead the market in vision, taking things which were good but raw, and rolling them into mature facilities which continues to make the computing industry grow!

Virtualizing Solaris

Abstract:
The movement from physical to virtual servers has been happening for decades. First, the use of Physical Domains under SPARC "big-iron" became possible when Sun purchased Cray SPARC assets in 1996 while SGI purchased the remaining. The Sun Enterprise 10000 was introduced in 1997 with Physical Domains. With the release of Solaris 10 almost a decade ago, physical systems could be moved to logical Zones under a single kernel in 2004. With the line of T processors, the ability to load multiple OS's on the same platform at the firmware layer became possible in 2006. This article discusses LDom's.

P2V:
Physical to Virtual Migration or P2V is possible to consolidate physical Solaris platforms onto various virtualized Solaris platform destinations - such as Zones, Branded Zones, or LDom's. The P2V process uses an archive called a FLAR.

P2V and Branded Zone:
An Oracle Enterprise Manager blog was published recently, explaining how to move a physical server to a Branded Zone. The May 15th blog was titled: "How to go Physical to Virtual with Oracle Solaris Zones using Enterprise Manager Ops Center."

Logical Domains:
The documentation describing the deployment of the Logical Domains for Oracle Enterprise Manager is available under Oracle's web site. Each Logical Domain, sits on top of the firmware of T-Class processors, and can host Solaris 11, Solaris 10, and Solaris 10 can host older Solaris 8 & 9 Operating System under Branded Zones.

Network Management Implications:
Network Management platforms can very easily be consolidated onto newer platforms, with very little effort, using free drag-and-drop tools such as Oracle Enterprise Manager. If a network management center is still running under multiple older Physical platforms, one should consider Zones or LDom's, which offer virtually no overhead (in comparison to systems such as VMWare or HyperV which require a foreign software layer between their domains and the hardware, introducing problematic latency under heavy loads.)

Oracle: Solaris 10 Update 11 Released!

Oracle: Solaris 10 Update 11 Released!

Abstract:
Solaris 10 was launched in 2005, with ground-breaking features like: DTrace, SMF (Services), Zones, LDom's, and later ZFS. The latest, and perhaps last, update of Solaris 10 was expected in 2012, to co-inside with an early release of the SPARC T5. In 2013, Oracle released yet another update, suggesting the T5 is close to release. The latest installment of Solaris 10 is referred to as 01/13 release, for January 2013, appears to be the final SVR4 Solaris release, with expected normal Oracle support extending to 2018. Many serious administrators will refer to this release as Solaris 10 Update 11.

(Oracle SPARC & Solaris Road Map, 2013-02-11)

What's New?
Oracle released the "Oracle Solaris 10 1/13 What's New" document, outlining some of the included features. The arrangement of the categories seems odd, in some cases, so a few were merged/re-orded below. Some of the interesting features include:



(Solaris 10 Update 11 Network File System Install Media Option)



• Installation Enhancements
  iSCSI LUN boot support
  NFS (Network File System) media support
  SVR4 Package Dependencies with Installer Preserve Option
  LU (Live Update) preserves Dump slice
  LU Pre-Flight Checker

(Solaris 10 Update 11 SVR4 Package Dependency Install Support)

• Administration Enhancements
  OCM (Oracle Configuration Manager) Client Service
  Oracle Zones Pre-Flight Checker
  SVR4 pkgdep (Package Depends) Command
  Intel x86 FMA (Fault Management Architecture) Sandy Bridge EP Enhancements
  AMD MCA (Machine Check Architecture) Support for Family 15h, 0Fh, 10h

# zfs help                                                              
The following commands are supported:                                   
allow       clone       create      destroy     diff        get         
groupspace  help        hold        holds       inherit     list        
mount       promote     receive     release     rename      rollback    
send        set         share       snapshot    unallow     unmount     
unshare     upgrade     userspace                                  

(Solaris 10 Update 10 zfs help system enhancements)

• Security Enhancements
  64 bit OpenSSL Support
  RESTRICTIVE_LOCKING now optional
  Speed Tuning/Improvements for: SSH, SCP, SFTP

# zpool help                                                            
The following commands are supported:                                   
add      attach   clear    create   destroy  detach   export   get      
help     history  import   iostat   list     offline  online   remove   
replace  scrub    set      split    status   upgrade                    
# zfs help create                                                       
usage:                                                                  
             create [-p] [-o property=value] ...                        
             create [-ps] [-b blocksize] [-o property=value] ... -V     

(Solaris 10 Update 10 zpool help system enhancements)

• ZFS File System and Storage Enhancements
  Help tiered into sub-commands for: zfs, zpool
  ZFS aclmode enhancements
  ZFS diff enhancements
  ZFS snap alias for snapshot
  Intel x86 SATA (Serial ATA) support for ATA Pass-Through Commands
  AMD x86 XOP and FMA Support
  SPARC T4 CRC32c Acceleration for iSCSI
  Xen XDF (Virtual Block Device Driver) for x86 Oracle VM

# zfs help create                                                       
usage:                                                                  
             create [-p] [-o property=value] ...                        
             create [-ps] [-b blocksize] [-o property=value] ... -V     

(Solaris 10 Update 10 zpool help create system enhancements)

• Freeware Enhancements
  Evince,  GNU Make, GNU gettext, GNU IDN (International Domain Naming), Ghostscript, gzip, Jakarta Tomcat, Lightning, rsync, Samba, Sendmail, Thunderbird, Firefox, wxWidgets
• New and Updated Device Drivers
  Broadcom BNXE (BCM57712 NetXtreme II 10 Gigabit Ethernet) Driver new support
  Intel SR-IOV (Single Root I/O Virtualization) support for i350 Gigabit and x540 10 Gigabit
  Sun Blade 6000 Virtualized 40 GbE NEM (Network Express Module)
  USB 3.0 Support (Block File Throughput: 150MBps Intel/AMD; 50MBps SPARC)
• Virtualization Enhancements
  LDom / Oracle VM SPARC Memory Dynamic Reconfiguration (DR) on Migration

Competitive Pressures:
Competition makes the Operating System market healthy! Let's look at the competitive landscape.

(Illumos Logo)

Solaris USB 3.0 is in a better support position than Illumos still missing USB 3.0 today since Solaris 10, Solaris 11, and Illumos all have top-of-the-line read and write flash accelerators for hard disk storage... a USB 3.0 flash cache will provide a nice inexpensive performance boost! Slower Solaris USB 3.0 support from 2013q1 on SPARC will be shunned with Solaris ZFS SMB's considering Apple MacOSX. Apple released USB 3.0 support in 2012q4 with Fusion Drive, making OSX a strong contender. Apple may have been late to Flash when proper licensing could not be agreed between Sun/Oracle and Apple, Apple is still late with deduplication, but now Oracle and Illumos are late with USB 3.0 to combine with ZFS.

(Lustre logo, courtesy hpcwire)

Sun purchased Lustre, for ZFS integration back in 2007. NetMgmt salivated as Lustre for ZFS was on-tap back in 2009, ZFS needed cluster/replication for a long time. Redhat purchased GlusterFS in 2011 and went beta in 2012, for production quality filesystem clustering. IBM released ZFS and Luster on their own hardware & Linux OS. NetMgt noted Lustre on EMC was hitting in 2012, questioned Oracle's sluggishness, and begged for an Illumos rescue. Even Microsoft "got it" when Windows 2012 bundled: dedupe, clustering, iSCSI, SMB, and NFS. It seems Apple, Oracle, and Illumos are the last major vendors - late with native file system clustering... although Apple is not pretending to play on the Server field.

(Superspeed USB 3.0 logo, courtesy usb3-thunderbolt.com)

The lack of File System Clustering in the final update of Solaris 10 is miserable, especially after various Lustre patches made it into ZFS years ago. Perhaps Oracle is waiting for a Solaris 11 update for clustering??? The lack of focus by Illumos on clustering and USB 3.0 makes me wonder whether or not their core supporters (embedded storage and cloud provider) really understand how big of a hole they have. An embedded storage provider, should would want USB 3.0 for external disks and clustering for geographically dispersed storage  their check-list. A cloud provider should would want geographically dispersed clustering, at the least.

(KVM is bundled into Joyent SmartOS, as well as Linux)

Missing native ZFS clustering and hypervisor at Oracle is making Solaris look "long in the tooth". Xen on Oracle Linux with Xen being removed from Solaris is a poor excuse by Oracle. Joyent's SmartOS KVM integrated into Illumos helps the Solaris community move forward, but what is the use of a hypervisor without shared-nothing clustered storage, to migrate those VM's at will? Missing USB 3.0 and native ZFS clustering is putting pressure on Illumos to differentiate itself in the storage market.

Conclusions:
Oracle Solaris 10 is alive and well - GO GET Update 11!!! Some of the most important features include the enhancements to CPU architecture (is SPARC T5 silently supported, since T5 has been in-test since end of 2013?), USB 3.0, iSCSI support for root disk installations, install SVR4 package dependency support, and NFS media support. Many of these features will be welcomed by SMB's (small to medium sized businesses.)

(Bullet Train, courtesy gojapango)

The Solaris Train continues to move at Oracle, producing high quality product, SPARC support, and new drivers (i.e. USB 3.0) - if Solaris 11, Illumos, or SmartOS releases ZFS clustering, the resulting OS will be market leading.

Remember The Past, Forging The Future

Abstract:
There is value to remember history and listen to recent great thinkers when considering the future. The following are my highlights of the presentation by Bryan M. Cantrill of Joyent from a little over a half-year ago.

USENIX 2011 Presentation - Fork Yeah! The Rise and Development of illumos

• Introduction of History from SunOS, Solaris, to Illumos - 0:00-1:20
• SunOS 4.x , Solaris 2.0, Solaris 2.1, - 1:20-3:00
• Source Code Control: NSE, NSElite, Bitkeeper, TeamWare, to The Creation of GIT 3:00-4:00
• Solaris 2.3, Solaris 2.4, Solaris 2.5 (almost killed Solaris for BSD) - 4:00-5:20
• BDFL Software Model (Benevolent Dictator For Life) - 5:20-6:00
• The Bonwick Youth - 6:40-7:00; 8:41-10:00
• The Impact of Windows Scalability Day 1997 - 8:00-8:41
• Revolutionary Ideas 2001 - 10:00-12:00
• 1) ZFS - 12:00-12:15
• 2) DTrace - 12:15-13:00
• 3) Kevlar/Zones - 13:00-14:00
• 4-8) FMA, SMF, FireEngine, Crossbow, Least Privilege- 14:45-15:44
• Sun: Feuding Bands of Warlords - 15:44-16:00
• Innovation from Engineers, not Management, Marketing, Customer - 16:45-18:20
• Kiln of unspeakable pain - 17:30-18:00
• People, Not Organizations, Innovate - 18:00-18:20
• OS is a Loss Leader - 18:40-19:00
• Sun was The Open Systems Company (ex. NFS) - 19:40-20:00
• The Open Sourcing of Solaris - 20:00-21:00
• The Crap Code, not Innovative, costs money - 21:00-21:20
• The First Open Source, DTrace - 22:00-22:30
• Proprietary Drivers, CDDL Licensing - 23:15-25:00
• Fork-a-phobic, Boards, Elections, Governance, Politics - 27:00-28:00
• Copyright Assignment Concerns - 29:00-29:30
• OpenSolaris Missile Crisis, independent OGB - 30:00-33:00
• End of an Era, Oracle bought Sun - 33:00-34:00
  Kicked butt, had fun, didn't cheat, loved customers, changed computing forever
• Birth of Illumos - 39:50-41:20
• 2010 Friday 13th memo, Death of OpenSolaris - 41:20-43:50
• Solaris Diaspora - 44:00-45:30
• Illumos Innovation, No Illumos Copyright - 45:30-46:50
• Joint Working Group for ZFS - 47:00-50:00
  Feature flag, compression ratio, estimated zfs send/receive, Zone I/O throttling, ZFS unmap, background destroy, resumable send
• DTrace - 50:00-52:00
  Log-linear quantization, KVM support for vmregs, tracemem(), toupper(), tolower(), etc.
• Zones - 52:00-55:00
  More multi-tenancy under Illumos than Solaris, svcs -Z, svcs -L, per-zone kstat, new rcapd,
• KVM - 55:00-57:00
  Windows, Linux, BSD on near bare-metal speed (ZFS, Dtrace, Zones); KVM in a Zone
• Black Hat: "Break out of QEMU into a more secure cell" - 57:00-57:17
• Illumos Distributions - 57:30-58:00
• Illumos Community Values - 58:00-58:30
  Freedom to Fork; Benevolent Oligarchy; Value Utility
  "We reject: kings, presidents, voting.We believe in: rough concensus, running code" 

Foot Note:
History has consequences - not long after this, Solaris 11 source code was leaked from Oracle... of which, ironically, no one wants to use. Open Sourced Components bundled with Solaris 11 can be found here.

Network Management at EMC World 2012

 

[EMC World 2012 Man - courtesy: computerworld]

Network Management at EMC World 2012

Abstract:
EMC purchase network management vendor SMARTS with their InCharge suite, a number of years ago, rebranding the suite as Ionix. EMC purchased Voyence, rebranding it as NCM (Network Configuration Manager). After EMC World 2012, they completed the acquisition of Watch4Net APG (Advanced Performance Grapher.) The suite of these platforms is now being rolled into a single new brand called EMC IT Operations Intelligence. EMC World 2012 was poised to advertize the new branding in a significant way.

Result:
EMC World 2012 in Las Vegas, Nevada was unfortunately pretty uneventful for service providers. Why was it uneventful?

The labs for EMC IT Operations Intelligence did not function. There were a lot of other labs, which functioned, but not the Network Management labs. EMC World 2012 was a sure "shot-in-the-head" for demonstrating, to service providers, the benefits of running EMC Network Management tools in a VM.

After 7 days, EMC could not get their IT Operations Intelligence Network Management Suite running in a VMWare VM.

Background:
Small customers may host their network management tools in a VMWare VM. Enterprises will occasionally implement their network management systems on smaller systems, where they know they will get deterministic behavior from the underlying platform.

Service Providers traditionally run their mission critical network management systems on larger UNIX Systems, so as to provide instant scalability (swap in CPU boards) and 99.999 availability (reboot once-a-year, whether they need to or not.)

The platform of choice in the Service Provider market for scalable Network Management platforms has been SPARC Solaris, for decades... clearly, for a reason. This was demonstrated well at EMC World 2012.

The Problem:
Why not host a network management platform in a VMWare infrastructure? Besides, the fact that EMC could not make it happen, after 1 year of preparation, and 7 days of struggling... there are basic logistics.

Network Management is dependent upon ICMP and SNMP.  Both of these protocols are "connectionless protocols" - sometimes referred to as "unreliable protocols". Why would a network management platform use "unreliable protocols"?

The IETF understands that network management should always be light (each poll is a single packet, while a TCP protocol requires a 3-way handshake to start the transaction, poll the single packet, then break down with another 3-way handshake. Imagine doing this for thousands of devices every x seconds - not very light-weight, not very smart. A "connection based protocol" will also hide the nature of an unreliable underlying network, which is what a network management platform is supposed to expose - so it can be fixed.

Now stick a network management platform in a VM, where the network connection from the VM (holding an operating system, with a TCP/IP stack), going down through the hypervisor (which is another operating system, with another TCP/IP stack, which is also sharing the resources of that VM with other VM's.) If there is the slightest glitch in the VM or the hypervisor, which may cause the the packets to be queued or dropped - the actual VMWare infrastructure will signal to the Network Management Centers that there is a network problem, in their customer's network!

Politics:
Clearly, someone at EMC does not understand Network Management, nor do they understand Managed Service Providers.

The Network Management Platform MUST BE ROCK SOLID, so the Network Operations Center personnel will NEVER mistake a alerts in their console from a customer's managed device as a local performance issue in their VM.

With EMC using Solaris to reach into the Telco Data Centers,  EMC later using Cisco to reach into the Telco Data Centers - EMC is done using their partners. VMWare was the platform of choice, to [not] demonstrate their Network Management tools on. Cisco was the [soon to be replaced] platform of choice, since EMC announced they will start building their own servers.

Either someone at EMC is sleeping-at-the-wheel or they need to get a spine to support their customers. Either way, this does not bode well for EMC as a provider of software solutions for service providers.

 

Business Requirements:
In order for a real service provider to reliably run a real network management system in a virtualized environment:

• The virtualized platform must not insert any overhead.
• All resources provided must be deterministic.
• Patches are installed while the system is live.
• Engagement of patches must be deterministic.
• Patch engagement must be fast.
• Rollback of patches must be deterministic.
• Patch rollback must be fast.
• Availability must be 99.999.  

Solutions:
There are many platforms which fulfill these basic business requirements, but none of them are VMWare. Ironically, only SPARC Solaris platform is currently supported by EMC for IT Operations Intelligence, EMC does not support SPARC Solaris under VMWare, and EMC chose not to demonstrate their Network Management suite under a platform which meets service provider requirements.

Today, Zones is about the only virtualized technology which offers 0%-overhead virtualizataion. (Actually, on SMP systems, virtualizing via Zones can increase application throughput, if Zones are partitioned by CPU board.) Zones, to work in this environment, seem to work best with external storage providers, like EMC.

Any platform which offers 0% virtualization penalty with ZFS support can easily meet service providers technical platform business requirements. Of these, the top 3 are probably the best supported by commercial interests

• Oracle SPARC Solaris
• Oracle Intel Solaris
• Joyent SMART OS
• OpenIndiana
• Illumian
• BeleniX
• SchilliX
• StormOS

Conclusion:
Today's market is becoming more proprietary each passing day. The movement towards supporting applications only under proprietary solutions (such as VMWare) has demonstrated it's risk during EMC World 2012. A network management provider would not be well advised to use any network management tool which is bound to a single proprietary platform element and does not support POSIX platforms.

Articles of Interest: 06-04-2012

An interesting set of articles tracing progress in ZFS, Solaris, and other OpenSolaris based distributions.

• Larry Wake posted a note for people wishing to quickly find articles on the Oracle blog site about Oracle technologies. Articles tagged as Solaris, ZFS, IPS, DTrace, Zones, Studio, Cluster.


• Chris Mellor from The Register wrote an article about a new ZFS storage appliance Zebi by Tegile.


• The Art of Virtualization has a recent article regarding P2V (Physical to Virtual) Solaris migration.


• Henkis of the SPARCv9 blog discusses the Illumian distribution release, based upon Illumos.


• Wim Coekaert posted a short article titled "The case for Oracle VM", which discusses the pricing for Oracle VM without Oracle hardware support, as well as detailing how Oracle VM is free with Oracle Hardware Support.


• ZFS comes of age, with Open Source community adding features not bundled by Oracle. Async Destroy of filesystems is being implemented under Delphix with Feature Flags, to enable multiple ZFS development communities to add features without stepping upon one another.


• Async Destroy being integrated into Illumos by Delphix Chris Siden.


• Feature Flag being integrated into Illumos by Delphix Chris Siden.


• Chris Siden from Delphix discussed "ZTest" - a ZFS backwards-compatibility testing suite in early 2012.

Oracle Solaris 11: Session 3: Why We Moved to Oracle Solaris 11

Oracle Solaris 11: Session 3: Customers and Partners: Why We Moved to Oracle Solaris 11

Online Forum
Oracle Solaris 11:
What's New Since the Launch
April 2012

Abstract:
Oracle released a series of 4 sessions on Oracle Solaris 11: What's New Since the Launch (see Session 1 and Session 2.) Various customers described by they chose to implement under Solaris. Some of the more interesting customers made decisions to move from VMWare to Solaris Zones.

Viewbiquity: Moved From RAM to Solaris, ZFS, and SSD's
Viewbiquity ran their cloud on a system based upon adding RAM to their platform as data scaled.With the move to ZFS, 4.5x data compression noted using ZFS, performance increased by merely adding SSD's, splitting mirrors across chassis (and rotate mirror off-site.)

BH Photo: Moving to Solaris 11
Immutable zones, Least Privilege for users in Zones, higher performance, regulatory requirements for one-function-per-server using zones, move zones between different hardware type (T or M) to resolve performance problems.

 

 

Oppenheimer: Moving from Windows/VMWare to Zones/Solaris
Reduced downtime for patching, reduced security footprint for account requirement using role-based administration, zones are invisible as far as storage and system capacity, 20 zones on a server.

 

Wells Fargo: Consolidation by Business Function
Reduce maintenance windows, patching with high-availability requires less technical resources (without splitting mirrors), 40K server environment with 40% human error downtime is being eliminated, zones for tier-1 applications for higher-availability, able to move zones directly into Solaris 11 environment seamlessly.

ULM University: Patch, Availability, and Live Debugging with DTrace
Patch management is easy with ZFS taking snapshot (allowing for rollback.) Patching takes a minute with fast reboot. DTrace is one of the most "outstanding tools of my career", outstanding uptime, makes administrators more relaxed.

MSC Software: Solaris 11 and Certification of Applications in Partner Remote Lab
Scheduled to do performance testing in the summer of SPARC SuperCluster. Oracle Solaris Remote Lab, to certify their applications. Oracle Partner Lab provides zones for customers to get qualified easily under Solaris 11. Virtual machines using latest solaris, database, and middleware - completely pre-configured, taking minutes to turn up for partners.

SAS: Solaris OS Differentiates with ZFS, Zones, and DTrace
Long time partner, SPARC in 1993, x86 in 2006. 30 Million lines of portable code between Intel and SPARC. Excellent relationship with Oracle Solaris Studio for development. ZFS, Zones, and DTrace differentiate product from all other vendors on the market. High performance without tuning. Guaranteed binary compatibility is an advantage as a cost savings. The November 1996 release of SAS still works, when loaded under Solaris 11.