Monday, October 7, 2019

Solaris 11.4: Eliminating Silent Data Corruption

Solaris 11.4: Eliminating Silent Data Corruption


Storage has been increasing in geometric proportions, for decades. As storage has been increasing, a problem referred to as Silent Data Corruption has been noticed. Forward thinking engineers at Sun Microsystems had created ZFS to manage this risk by having discovery & correction occur passively & automatically upon future reads & writes. Oracle later purchased Sun Microsystems and introduced proactive automated discovery & correction on a monthly basis, as part of Solaris 11.4

The Problem:

Silent Data Corruption has been measured by various industry players dealing with massive quantity of storage.
the fast database at Greenplum, which is a database software company specializing in large-scale data warehousing and analytics, faces silent corruption every 15 minutes.[9] As another example, a real-life study performed by NetApp on more than 1.5 million HDDs over 41 months found more than 400,000 silent data corruptions, out of which more than 30,000 were not detected by the hardware RAID controller. Another study, performed by CERN over six months and involving about 97 petabytes of data, found that about 128 megabytes of data became permanently corrupted.
 As storage continues to expand, the need to resolve silent corruption became more important.

The Passive Solution:

Jeff Bonwick at Sun Microsystems created ZFS, specifically to address storage as data storage quantities increased. The ZFS File System was not a 32 bit File System, like 30 year old technology, but was engineered to be a 128 bit filesystem, projected to accommodate data into the next 30 years. With such  a massive quantity of data to be retained, Silent Data Corruption was addressed by performing a checksum on the data during the write and verifying it on future reads. If the checksum does not match on the read, then a redundant block of the data on the ZFS File System will be automatically read, and a correction would occur to the formerly read bad block. This feature was very unique to Solaris.

A system administrator can read every block via an operation referred to as a "scrub".
sc25client01/root# zpool list rpool
rpool  416G   296G  120G  71%  1.00x  ONLINE  -

zpool scrub rpool 

This scrub will continue in the background until all disks had all of the blocks read. The scrub always reads data at a rate which does not interfere with the operation of the platform or applications.

The Proactive Solution:

With the release of Solaris 11.4, formerly known as Solaris 12, an automated schedule of reading every byte of data in the entire pool is scheduled by default in the storage pool once a month. By reading every block of data once a month, silent data corruption can be rooted out and corrected automatically, which is a very unique feature of Oracle's Solaris!

Under an older OS release (Solaris 11.3 SRU 31),  notice that the property does not exist.
sc25client01/root# uname -a
SunOS sc01client01 5.11 11.3 sun4v sparc sun4v

sc25client01/root# pkg list entire
NAME (PUBLISHER) VERSION                    IFO
entire           0.5.11-    i--

sc25client01/root# zpool get lastscrub rpool
bad property list: invalid property 'lastscrub'
For more info, run: zpool help get
Under a modern OS release (Solaris 11.4 SRU 13), the last scrub occurred less than a month ago.
sun9781/root# uname -a
SunOS sun1824-cd 5.11 sun4v sparc sun4v

sun9781/root# pkg list entire
NAME (PUBLISHER) VERSION                    IFO
entire           11.4-       i--

sun9781/root# zpool get lastscrub rpool
rpool  lastscrub  Sep_10  local
The last scrub details can be seen through the status option.
sun9781/root# zpool list
rpool  278G  36.9G  241G  13%  1.00x  ONLINE  -

sun9781/root# zpool status
  pool: rpool
 state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
        pool will no longer be accessible on older software versions.
  scan: scrub repaired 0 in 16m24s with 0 errors on Tue Sep 10 03:42:44 2019

        NAME                       STATE      READ WRITE CKSUM
        rpool                      ONLINE        0     0     0
          mirror-0                 ONLINE        0     0     0
            c0t5000CCA0251CF0F0d0  ONLINE        0     0     0
            c0t5000CCA0251E4BC8d0  ONLINE        0     0     0

errors: No known data errors
The above 278 Gigabyte pool was able to be read in a little over 15 minutes, and checked with no errors to be corrected.


Network Management is well aware that the more storage that is needed that the more critical the data recovery process becomes. Redundancy through advanced file systems like ZFS under managed services class operating systems like Solaris are a good choice. Solaris 11.4 keeps data healthy, no matter what quantity of physical disks managed or data being retained.

Friday, September 20, 2019

Solaris 10: Extended Support to 2024

Solaris 10: Extended Support to 2024

Solaris 10: Introduction

Oracle Solaris 10 has been an amazing OS update, including ground breaking features like Zones (Solaris Containers), ZFS, Services, Dynamic Tracing (against live production operating systems without impact), and Logical Domains. These features have been emulated by the market (imitation is the finest form of flattery!)

Solaris 10: End of Life

As with all good things, they must come to an end. Sun Microsystems was purchased by Oracle and eventually, the greatest OS known to the industry needed to be updated. Oracle set a retirement date of January 2021. Oracle had indicated an uplift in support costs would be needed, for Solaris 10 systems.

Solaris 10: Extended Support to 2024

No migration tools were ever provided by Oracle to facilitate migration from Solaris 10 to Solaris 11, so migration to Solaris has been slow. Oracle had decided in September 2019 that Extended Support for Solaris 10, without additional financial penalty, would be delayed to 2024!

Saturday, September 7, 2019

Creating an SFTP Server

Creating an SFTP Server


In the early days of the internet, the file transfer mode of choice was FTP. This had remained the internet standard for many years, until the need for encryption  had become so prevalent. SFTP happens to be one option for delivering files.


A good reference on creating an SFTP server can be see on this web site. This is not isolated to Linux, but can be used under SVR4 & POSIX systems like Solaris, or other Open Sourced based operating systems based upon OpenSolaris or Illumos.


Usually if SFTP is all that is desired to be granted, one should not allow standard TTY based logins. One option is to set up a shell using the "nologin" binary. A quick reference on using "nologin" vs other binaries like "false". In early UNIX days, "sync" was sometimes used!

Saturday, August 31, 2019

Packaging: OpenSolaris, Solaris 11 & Illumos

Packaging: OpenSolaris, Solaris 11 & Illumos

Packaging under UNIX Systems had traditionally been the SRV4 type of packaging. Under newer Operating Systems, network enabling the packaging had been accomplished, through proprietary extensions. Open Source Communities, like OpenCSW, had created network enablement for SVR4 packages. Illumos had done a similar move, from the OpenSolaris base. This tool is called "pkg".

What is "pkg"

The "pkg" tool is called the Image Packaging Systems. Today, packaging under the direction of Oracle had moved to Python language based packaging, instead of native C based binaries, placing greater levels of external language dependencies on the base OS. SVR4 packaging uses multiple binaries (i.e. pkgadd, pkgrm, pkgproto, etc.), while "pkg" bundles many aspects of packaging into a single executable.

Cheat Sheet & Exercises

A short Image Packaging System cheat-sheet is available from Oracle on the , using "wireshark" as an example. The cheat-sheet can be used for developing other exercises to download & manage other tools like: fping & top

Wednesday, July 31, 2019

ZombieLoad Vulnerability: SPARC Solaris Immune

[ ZombieLoad Logo, courtesy ]

ZombieLoad Vulnerability: SPARC Solaris Immune


Computing platforms have long had issues with MalWare, dating back to the MS-DOS days. Windows systems had been targeted, due to their ubiquity as well as the ability to install software with no user interaction into the system using pre-bundled keys. For the most part, UNIX Systems have been immune to most malware. A new classification of malware had become apparent, using CPU vulnerabilities, normally related to the Intel processor (leaving SPARC processors immune.) The latest vulnerability is ZomieLoad, affecting Intel processors and non-UNIX platforms such as Linux and Windows.

What is ZombieLoad

A new family of vulnerabilities on the Intel Processor have become apparent in Mid-May 2019. As an aggregate, they are referred to as "Microarchitectural Data Sampling" or MDS vulnerabilities. ZombieLoad is one of these vulnerabilities. Oracle provided a nice list of CVE's with summary details: CVE-2019-11091, CVE-2018-12126, CVE-2018-12130, and CVE-2018-12127
  • CVE-2019-11091: Microarchitectural Data Sampling Uncacheable Memory (MDSUM)
  • CVE-2018-12126: Microarchitectural Store Buffer Data Sampling (MSBDS) 
  • CVE-2018-12127: Microarchitectural Load Port Data Sampling (MLPDS)
  • CVE-2018-12130: Microarchitectural Fill Buffer Data Sampling (MFBDS)

  • How does it affect SPARC Solaris?

    If you return back t the Oracle provided CVE's above, you will notice the following 2x lines:

    Oracle Hardware

    •Oracle has determined that Oracle SPARC servers are not affected by these MDS vulnerabilities.


    Oracle Operating Systems (… Solaris) and Virtualization:

    •Oracle has determined that Oracle Solaris on SPARC is not affected by these MDS vulnerabilities.


    If you are fortunate enough to be running SPARC Solaris, you are immune again. If you are not on SPARC Solaris, but on a less secure Intel based Windows or Linux platform - well, you will be needing to supply your operating system vendor's CPU microarchitecture patch and probably reboot. Let's hope you are not having to roll-your-own fix.

    Monday, July 29, 2019

    NYLUG: Talk on ZFS on Linux

    NYLUG: Talk on ZFS on Linux


    Older file systems were based upon 32 bit UFS (UNIX File System) technologies, which lasted about 10 years, but started becoming tight with modern storage. Sun had produced a modern file system to last the next 10 years - it was called ZFS. ZFS is a 128 bit file system, created by Sun Microsystems, who was acquired by Oracle Corporation, is the primary active maintainer, and feature record of reference. ZFS was open sourced with OpenSolaris and other Open Source distributions started to use it, including Illumos. Eventually, Linux started to leverage ZFS. This talk is by Paul Zuchowski with a little bit of information regarding ZFS on Linux.

    New York Linux User Group:

    Paul Zuchowski is a former Sun Microsystems engineer. I became aware of him when he left a comment on a blog that I followed, He recently gave a talk at the NYLUG in April 2019 regarding the current state of ZFS on Linux. Many of the features, which are currently in the Oracle Solaris 11 release of ZFS are being actively worked on, in order to catch up, in the Linux Community.


    While some performance problems engineered by Oracle for Solaris based ZFS features may not even be a sparkle in the Linux community's eye, they are actively trying to find solutions. Hundreds of PB's of storage is currently contained in ZFS on Linux, just with a couple of companies, so clearly ZFS under Linux is mainstream enough for production use.

    Tuesday, June 4, 2019

    The Business Plan for Sun Microsystems

    [Sun Microsystems Logo]

    The Business Plan for Sun Microsystems

    Have you ever wondered what the business plan looked like for a tech startup which becomes a multi-billion dollar international corporation?

    Well, this is an example Business Plan for Sun Microsystems, from back in 1982!

    Tuesday, May 7, 2019

    Fujitsu SPARC M10 and M12 Physical Partitions

    [SPARC Logo, courtesy SPARC International]

    Fujitsu SPARC M10 and M12 Physical Partitions


    A conversation occurred recently, with a marketing expert regarding, whether an OS instance could span multiple chassis. At the time, this author indicated that it was not possible, but capabilities of the Fujitsu SPARC products were not well advertised. Fujitsu published a presentation describing the capabilities.
    [Solaris Logo, courtesy Oracle and Sun Microsystems]

    Oracle Solaris

    There is an off-the-shelf OS, which has the ability to span multiple chassis, but you have to have the right chassis. Fujitsu has long engineered high-end servers with mainframe reliability. Fujitsu had also supplied SPARC hardware for Solaris, from the days f the first SPARC processors! When engineered correctly, Solaris can span chassis.

     An example slide which shows a Physical Partition on individual chassis or spanning 2 chassis.
    [Fujitsu Logo, Courtesy Fujitsu]

    Building Blocks

    One may ask, how this physically works?
    Fujitsu offers a system with an expandable chassis, where there is an external crossbar, attaching multiple chassis together, similar to the way cards in a traditional Oracle chassis uses a crossbar inside the same chassis.

    An example slide of physical Building Blocks illustrated above.


    SPARC Solaris continues to have unusual features, which are not common in other equipment vendor equipment. Had this author been aware of this capability, cloud building may have been done a different way... providing a seamless way to expand clouds by merely adding one chassis at a time and never having to worry if an OS instance needs to grow large.

    Monday, March 18, 2019

    Oracle: OpsCenter 12.3.3

    Oracle: OpsCenter 12.3.3

    What is OpsCenter?

    Oracle support remote systems management tool, offered from Oracle for their hardware & OS's. It is available for gratis. It is referred to as Oracle Enterprise Manager OpsCenter.

    What does OpsCenter do?

    OpsCenter will provision OS's on bare-metal, as well as distribute patches via a GUI tool,

    What makes OpsCenter different from Oracle Enterprise Manager?

    OpsCenter will manage all the way down to the hardware ILOM, without an OS agent, something which Oracle Enterprise Manager does not do.

    Where can I get it?

    You can  download OpsCenter from Oracle's web site.

    What is the latest version?

    As of this writing, the latest version is 12.3.3 and requires a JIDR for the most recent Oracle Hardware and Operating System support.

    Monday, March 11, 2019

    Fujitsu: Run Solaris 10 & 11 Natively on New Bare Metal

    [Fujitsu Logo, courtesy Fujitsu Ltd.]

    Fujitsu: Run Solaris 10 & 11 Natively on New Bare Metal


    Sun Microsystems originally designed the SPARC processor and merged AT&T and BSD UNIX together to form Solaris. Fujitsu tarted developing clone hardware, which provided a second manufacturing source, fufilling military applications requirements. Oracle purchased Sun and later ended the native support of Solaris 10 on newer SPARC platforms. Fujitsu continues to support Solaris 10 & 11 on native Fujitsu SPARC M12 Platform.

    [Fujitsu SPARC64

    The M12

    In 2017, Fujitsu released the SPARC64 XII processor, reaching the fastest performance in the industry, of all processors in the market.  This processor was placed in a chassis named the M12. Unlike the newer Oracle chassis, these platforms can run native Solaris 10 or 11, without virtualization.

    This chassis comes in 2 flavors: M12-2 and M12-2S. The M12-2S is perhaps, the most interesting: the 2S can scale be adding up to a total of 12 chassis in a system to provide 32 sockets and support over 3000 threads by merely adding one chassis at a time!

    [Solaris Logo, courtesy of Sun Microsystems, now Oracle]

    Solaris 10

    It should be noted, Solaris 10 does have a definitive life expectancy. New features are not expected, as the OS is now in Extended Support. Extended support offered Solaris 10 Patch Clusters. April 17 in 2018 marked the first set of Extended Support Patches, in Classic Solaris. As of this publishing, Oracle released another set of Solaris 10 Patches in January 9, 2019. The details for most current Recommended Solaris 10 Patch Set can be found by following the link. The final set of Extended patches will be released in January 2021. There is an uplift for Solaris 10 Extended Support, while Solaris 11 is a free update... and this is preferable!


    While bare metal may be appealing to some applications, such as dedicated clustered solutions where redundancy is built at the application layer, most engineers prefer the portability of LDoms on a chassis cluster, where LDoms can be live migrated onto another chassis as planned maintenance is conducted on the drained chassis. The Solaris 10 bare metal support offered by Fujitsu provides large scale users, who desire bare metal performance the least amount of complexity, an option offered by no other SPARC vendor.

    Monday, March 4, 2019

    Solaris 11: Hardware Compatibility List - 2019q1

    Solaris 11: Hardware Compatibility List - 2019q1


    For those who are covering the life of real UNIX systems, a good place to track the progress of Solaris has been the Hardware Compatibility List. For the Q1 quarter of 2019, it may be helpful for readers to understand what the latest hardware is, that has been certified for Oracle Solaris, to execute 2019 purchases.

    [Oracle Logo, courtesy Oracle Corporation]

    Oracle's Submissions

    Oracle's most recent submission - SPARC M8-8 on 2018-09-07 for Solaris 11.3 & 11.4.

    Fujitsu's Submissions

    Fujitsu's most recent submission - SPARC M-12 on 2017-07-10 for Solaris 11.3 & 11.4.
    Interestingly enough, also certified is Solaris 10 1/13 [aka Solaris 10 Update 11]!

    [Dell Logo, courtesy Dell Corporation]

    Dell Additions

    Dell continues to submit hardware into the Hardware Compatibility List.
    They included 3x submissions:
    1. 2018-11-14 PowerEdge R640 - 2x socket, 8x cores/socket, Intel Bronze 3106 CPU @ 1.70GHz
    2. 2018-12-20 PowerEdge R840 - 4x socket, 4x cores/socket, Intel Gold 5122 CPU @ 3.60GHz
    3. 2019-01-15 PowerEdge R740 - 2x socket, 14x cores/socket, Intel Gold 5120 CPU @ 2.20GHz
    All of these were for Solaris 11.4.

    The Odd Man Out

    There is a Chinese outsourcing company which also appears on the HCL, called Inspur.
    They have 2x submissions:
    1. 2018-11-08 NF5280M5 - 2x socket 8x core/socket, Intel Silver 4110
    2. 2018-11-15 NF5180M5 - 2x socket, 24x core/socket, Intel Platinum 8176
     Inspur was certifying for Solaris 11.4.

    Tuesday, February 26, 2019

    Oracle: Un-Publishes SPARC Solaris Roadmap?

    Oracle: Un-Publishes SPARC Solaris Roadmap?

    It has been the tradition of Oracle to maintain a public roadmap for the SPARC Solaris product line, since  the purchase of Sun Microsystems.


    As of 2018, Oracle had been promoting a roadmap which illustrated a 2020 SPARC M8+.

    Fujitsu Today

    Fujitsu is still promoting, in 2019 on a roadmap, a new SPARC Server in 2020, as of Feb 25, 2019.

    Oracle Today

    The "Assets" link was previously available. The "Servers-Storage" link was also previously available. Today, they are no longer available. They had disappears, approximately in January 2019. Oracle's 2018 roadmap was available from  around May 25th.

    It should be noted, Oracle's web site had gone through a re-design, with different base URL's now being available, so this could have just been an oversight.


    Network Management had traditionally not listened to rumors in the industry, regarding layoffs, but had traditionally depended upon hard evidence such as documents, hiring announcements, code releases, etc. There is a degree of concern, in the industry, what Oracle intends on doing with SPARC & Solaris. Oracle continues to release updates for hardware management software, like OpsCenter, and Fujitsu appears to be holding on to it's roadmap.

    Sunday, February 24, 2019

    DoD: Security Technical Implementation Guides (STIGs)


    The US Department of Defense [DoD] released a document in regard to issued instruction 6500.01 which declared Defense Information Systems Agency [DISA] "develops and maintains control... security technical implementation guides (STIGs)... that implement and are consistent with DoD cybersecurity policies, standards, architectures, security controls, and validation procedures, with the support of the NA/CSS, using input from stakeholders". This resulted in the release of such documents.


    Security demands the rigorous standards which must abide by, uniformly, since security is only as good as it's weakest link. The DoD offers Oracle Solaris UNIX Operating System Security Technical Implementation Guides, for SPARC & Intel, as well as for Solaris 10 & 11. 

    [html] Index
    [zip] STIG - Solaris 10 SPARC Version  1 Release 24
    [zip] STIG - Solaris 10 x86 Version  1 Release 24
    [zip] STIG - Solaris 10 SPARC Version  1 Release 16
    [zip] STIG - Solaris 10 x86 Version  1 Release 16


    Solaris, for both SPARC and Intel Platforms, continue to be the safest platforms for implementing systems on, in the world. It should be noted that Solaris 10 is fast approaching end-of-life, therefore Solaris 11 should be considered a primary platform for implementation.

    Wednesday, February 6, 2019

    SPARC: Lift & Shift Solaris 10 ZFS RPool

    SPARC: Lift & Shift Solaris 10 ZFS RPool


    Since the adoption of SMP (Symmetric Multi Processing) by Sun Microsystems, there has been a shift in consolidation on chassis: co-hosting applications on a common OS instance, applications leveraging limited protections with "chroot", hosting multiple OS's on the same chassis through PDoms (Physical Domains), hosting multiple OS's via LDoms (Logical Domains), hosting applications in Solaris Containers (Zones). Sun had created techniques to lift-and-shift UFS based Solaris instances to LDoms, but had not provided a way forward for virtualization of ZFS bsed Solaris 10 systems... until days ago.

    Solaris 10 P2V With ZFS Root

    With Solaris 10 coming "end of life" and the push to Solaris 11 going strong, there is still a need to consolidate older model chassis. There is a guide called "Lift and Shift". Sekhar Lakkapragada, Senior Principal Software Engineer at Oracle, wrote an article on  how to accomplish this very task:


    Sekhar posted a tool to accomplish this very activity, to the Oracle Solaris Blog. A short synopsis:
    We are happy to announce that we now offer a new tool called ldmp2vz(1M) that can migrate Oracle Solaris 10 source systems with ZFS root. This tool has two commands; ldmp2vz_collect(1M) and ldmp2vz_convert(1M). These two commands enable migration of a sun4u or a sun4v Oracle Solaris 10 physical machine to an Oracle Solaris 10 guest domain on a system running Oracle Solaris 11 in the control domain as shown in this diagram.
    In short, it is a 3 step process, not dissimilar from the ldmp2v tool:
    1. Collection: Run ldmp2vz_collect(1M) on the source for system image and configuration
    2. Preparation:  Create a new Oracle Solaris 10 SPARC guest domain using ovmtdeploy(8), Jumpstart, or DVD.
    3.  Conversion: Run ldmp2vz_convert(1M) from prepared Solaris 10 guest domain using Live Upgrade technology.


    This has been a long time coming, but it is welcome, none the less.

    Friday, January 11, 2019

    ZFS Primer for Solaris 11.3

    ZFS Primer for Solaris 11.3

    What is ZFS?

    ZFS is a flattened Volume Management & File System infrastructure that takes care of just about any basic OS needs. With 32 bit filesystems running out of steam, ZFS was created as a 128 bit filesystem to last for the

    What are the features?

    Basic features include:
    - basic file system
    - multiple file systems sharing a single pool of storage
    - concatenation/striping to extend pools of storage
    - mirroring to protect pools of storage
    - hot-sparing of storage
    - RAID to affordably protect a pool of storage (should use battery backup)
    - RAIDZ to affordably protect a pool of storage (without battery backup)
    - automatic silent data corruption correction
    - on-line silent data corruption search & correction
    - double Parity RAID to survive a dual disk failure
    - snapshot, for read-only point-in-time data consistency
    - scheduled snapshots, to provide rollback from user level data corruption 
    - diff, to determine differences between snapshots
    - clones, to make snapshots read-write
    - promotion, to make a clone the record of reference
    - rollback, to restore a snapshot to become the record of reference
    - send, to backup a pool or filesystem to an alternate location
    - receive, to restore a pool or filesystem from an alternate location
    - deduplication, to make it an exquisite repository for massive VM repositories
    - compression, to speed I/O and store more data
    - encryption, to secure data
    - sharing storage over NFS natively
    - sharing storage over CIFS natively
    - sharing storage over iSCSI natively

    Is it Stable?

    It was introduced in 2005, very stable.

    Where is it used?

    It is used as the default file system for Solaris 11, newer Operating Systems, storage appliances, and was even introduced into older operating systems (like Solaris 10 or Linux) as optional root disk storage.

    Where can I find out more?

    In December 2016, Fujitsu released a document called the "ZFS Implementation and Operations Guide". It is about the best introduction to ZFS that this author had ever seen. It is well worth the read!

    Tuesday, January 8, 2019

    SPARC Solaris Upgrades Continue - US AirForce

    SPARC Solaris Upgrades Continue - US Air Force

    The US Air Force is contracting with Stellar Innovations & Solutions, Inc is contracting to upgrade Sun T15140 Chassis to T7-1 Chassis, across the nation!
    [Sun SPARC T5140]
    This is a full time job for about 4 engineers with experience. Engineers will unbox, install, wire, label, and make sure the project manager & customer are aware of the progress.

    [Oracle SPARC T7-1]
    This is a great opportunity to execute on your Solaris & SPARC Skills!