IBM: Sun Best in OS Vulnerabilites Reporting and Patching

IBM: Sun Best in OS Vulnerabilities Reporting and Patching - 2009-1H

I know what you are thinking, IBM thinks Sun outperformed the rest of the market in regards to OS security?

Apparently, in the 1st half of 2009, IBM commends Sun for security above all other competitors, even their own coders and product partners!

Sun is the best at sharing information about its operating system's vulnerabilities and patching them, reports IBM's "X-Force 2009 Mid-Year Trend and Risk Report." This analysis of various online threats and vulnerabilities examined statistics for the first half of 2009.

By what metrics did IBM measure?

Solaris had only 26 percent of the total number of OS vulnerabilities... Microsoft had the most ... with 39 percent of the total.

But this was not the only metric...

Sun's patch rate also was deemed impressive with only four percent left unpatched. "For the vast number of disclosures Sun makes, they have an impressive patch rate (only four percent left unpatched)"... The average patch rate within the industry is 49 percent. Sun's four percent rate tops Apple's 18 percent and Microsoft's 17 percent.

This is fairly eye opening to the industry - Sun clearly is better controlling their own destiny with Solaris than the competitors.

Apache: Hack, Rollback, Recover, and Secure

Apache: Hack, Rollback, Recover, and Secure

Apache: Hack

It was a bad day for the Apache team - apache.org was hacked 2009-08-28 through their CentOS Linux derivative. The CentOS is a Linux distribution bundled with RedHat Package Manager.



Apache: Rollback

When your system has been hacked, what would be your first choice to recover?

Go to an on-line backup? (How do you know it was not also compromised?)
Go to a tape backup? (How do you know it was not also compromised?)
How far back to you go? (Do you only keep 3 backups?)
Do you re-build from scratch?

aurora.apache.org runs Solaris 10, and we were able to restore the box to a known-good configuration by cloning and promoting a ZFS snapshot from a day before the CGI scripts were synced over. Doing so enabled us to bring the EU server back online, and to rapidly restore our main websites.

The Apache team was very fortunate - they implemented Sun SPARC Solaris ZFS. They were able to roll back to a snapshot and recover.

Apache: Recover

It was mentioned that the ZFS snapshot was used to ultimately recover the web site. How does ZFS offer this type of capability?

ZFS is a 128 bit unified file system and volume manager. It offers virtually unlimited volume size... and virtually unlimited snapshots. Any production environment exposed to the internet should use ZFS as a best practice, to be able to quickly resort back to a pre-corrupted stage.

For example, you can schedule a snapshot of your system every day, or every 15 minutes (if you want!), and you can hold these snapshots for a week, with virtually no overhead. At any point in time, you can drop back to a previous release, just as the apache.org foundation decided to do.

What exactly is a "snapshot"? The zfs manual page reads:

A read-only version of a file system or volume at a given point in time. It is specified as filesystem@name or volume@name.

The process of taking an old snapshot and making it writable is called a "clone". The manual page reads:

A clone is a writable volume or file system whose initial contents are the same as another dataset. As with snapshots, creating a clone is nearly instantaneous, and initially consumes no additional space.

This "clone" can be "promoted" so as to become the master version of the file system volume, erasing the old content. This is also described in the zfs manual page:

Clones can only be created from a snapshot. When a snapshot is cloned, it creates an implicit dependency between the parent and child. Even though the clone is created somewhere else in the dataset hierarchy, the original snapshot cannot be destroyed as long as a clone exists. The origin property exposes this dependency, and the destroy command lists any such dependencies, if they exist.
The clone parent-child dependency relationship can be reversed by using the promote subcommand. This causes the “origin” file system to become a clone of the specified file system, which makes it possible to destroy the file system that the clone was created from.

This is basically the process that Apache.org used to recover their Linux web servers.

Apache: Secure

Securing those web servers are a different story. SSH is no magic bullet - this was also compromised. There is no magic bullet in the open-source world. Different open-source communities have different certification processes. A Linux kernel may come up, be slowly accepted into a distribution, with patches made along the way from the original kernel team as well as a separate distribution company.

One of the sections was about positive lessons :

• The use of ZFS snapshots enabled us to restore the EU production web server to a known-good state.
• Redundant services in two locations allowed us to run services from an alternate location while continuing to work on the affected servers and services.
• A non-uniform set of compromised machines (Linux/CentOS i386, FreeBSD-7 amd_64, and Solaris 10 on sparc) made it difficult for the attackers to escalate privileges on multiple machines.

This is the "gold standard".

While your front-end Linux boxes may get hacked, diversifying your infrastructure with an additional (secure) OS, and using a real unified file system & volume management system like ZFS under Solaris makes the hackers struggle while provides options to every-day system administrators.

IBM recently acknowledged Sun Solaris as being best-in-class in security - something to keep in mind.

Be cautious of other vendors like Microsoft with long outstanding security holes in their IIS web serving software or security issues which they refuse to fix. These are not good candidates for customer facing systems - for obvious reasons. Imagine gaining access to the IIS server and just querying the usernames and passwords from the ebedded MSSQL server - oh, the humanity!

Future of Storage: Flash

Future of Storage: Flash

Abstract

With ever increasing storage size increases, cost decreases, and performance increases - it seems like Flash storage will soon be a winner in the Managed Services arena. Others are writing about this technology, such as StorageMojo and The Register. Understanding where the technology is going is a good start, but understanding your technological bottlenecks is required for application.

What's New?

The flash DIMM format is an opportunity to significantly change the computing industry for the long term. Pictured below is a Sun Flash DIMM, in the form factor of common laptop memory.


Since hard disk drives are normally always spinning and generating massive quantities of heat - failure rates were high. High failure rates necessitated easy access in arrays through front and rear chassis access. Removal of the heat generation and mechanical movement increases reliability - so why would one need to suck up space in the front or rear of a rack with the center of the rack mount unit being mostly empty?

DIMM is the way to go.

What's Up and Coming?

Sun Systems designer, Andy von Bechtolsheim discussed the use of flash a quarter ago, both in the regular market, as well as in the future of the Sun marketing products.

One example of a core building block was a 4 Terabyte 1U high storage unit.


While 4 Terabytes in a 1U high rack space may not make people jump for joy, the news is really around the benefits for a dramatic increase in performance. When one can increase performance 100x, use 1/100th the rack space, and use orders of magnitude less power - this can drive change in any business.

Sun briefly posted a PDF of the F5100 storage platform, but this document was pulled. Google still has the HTML version of the "Sun Storage F5100 Flash Array Getting Started Guide" document, although it is fairly stripped of visual content and structure. You can see from the HTML that it was published in July of 2009.

What's Here Today

What is very comforting is that this up & coming technology is already supported by standard storage management tools - Sun StorageTek Common Array Manager (CAM).


The F5100 Flash Array is steadily appearing in more standard Sun documentation, for example the "Sun StorageTek Common Array Manager User Guide for Open Systems". There is also a PDF of this guide (as well as others) available, for common consumption.

Application in Network Management

For Open Source polling & graphing software, the I/O becomes inhibitive in large installations. The need for many spindles in order to keep up with the data read & write rates had created the architectural need to split the database from the polling software. With massive quantities of data coming into a database on very rapid & regular polling rates - the need for archiving this data becomes increasingly important, but secondary to the performance since massive numbers of spindles will leave high capacity drives mostly empty.


What would happen if the artificial need for breaking storage away from the pollers disappeared?

The architecture could simplify and re-consolidate onto a single server with multiple [virtual] pollers. The result would be a reduced level of system complexity (fewer servers, switch ports, physical ports, drive interconnects, etc.), increase performance (eliminate the need for massive external storage), increase reliability (fewer moving mechanical parts), and overall decrease costs.

Microsoft IIS Vulnerabilities Across Releases

Microsoft IIS Vulnerabilities Across Releases

New IIS attacks (greatly) expand number of vulnerable servers

The Register published a short article of concern for those of us in the Network Management industry, where we customer or internet facing platforms for reporting delivery.

Microsoft continues to say that IIS5 running on Windows 2000 appears to be the only version that is vulnerable to attacks that can remotely execute malicious code on an underlying server. But it's now clear that hackers can target every version of IIS to cause denial-of-service attacks.

If you have a current or legacy IIS server - this may place your installation at risk. This is an piece of old code, meaning that historical code that you have not touched for awhile will be at risk. The risk centers around industry standard FTP protocol, one of the backbone protocols of the internet.

If Microsoft is not releasing patches for your old release of IIS, time to think about replacing that old portal.

Microsoft rejects call to fix SQL password-exposure risk

Microsoft rejects call to fix SQL password-exposure risk



Abstract

Most serious Managed Services Element Management Platforms, which depend on external databases, traditionally do not depend on databases such as Microsoft SQL. This article illustrates one of the reasons: security.

The Problem

"Applications go to great lengths to obfuscate passwords when they are needed within the software, and should not store passwords as 'clear text,' either in memory (as is the case with this vulnerability) or on disk," Sentrigo's advisory stated.

Microsoft has rejected the company's calls to change the way the software handles passwords, saying people with administrative rights already have complete control of the system anyway.

The Response

"Microsoft has thoroughly investigated claims of vulnerabilities in SQL Server and found that these are not product vulnerabilities requiring Microsoft to issue a security update," a spokesman wrote in an email. "An attacker who has administrative rights already has complete control of the system and can install programs; view, change, or delete data; or create new accounts with full user rights."

What this means to Network Management

The problem with passwords being stored in the clear is not that an infected system could have data destroyed on it, but rather other systems what work with that infected system could be infected!

Of course, behaviors like this are rampant with Day-0 Exploits, Microsoft SQL Worms, Microsoft Windows Viruses, etc. Another place to get passwords by malware is just another reason not to implement such a system in an area where customer managed devices are routable.

If a system is storing passwords for thousands of managed systems in the clear, an infection of a central system could be disastrous for the managed customer edge devices.

A developer in a company may have the option to secure passwords or not - but if the developer in a company ever has to meet a PCI audit and the vendor does not offer that option, then the company providing the managed services is placed in tremendous risk.

Multi-Node Cluster Shared Nothing Storage

Multi-Node Cluster Shared Nothing Storage

Abstract

A number of months back, a new release of Sun Cluster was released, in conjunction with OpenSolaris 2009.06. This release offered a new architecture for a lower cost fail-over cluster capability using Shared-Nothing Storage. This paper discussed the benefits using a broader implementation plan to further reduce costs and increase scalability.

Shared Nothing Storage

With the advent of ZFS under Solaris and ComStar under OpenSolaris, there is a new no-cost architecture in the world of high-availability under Sun - Shared Nothing Storage.

The benefits are clear in this environment:

• External Storage is not required (with it's complexity and costs)


• Additional storage area network infrastructure is not required (with it's complexity and costs)


• The OS of the active node continually keeps all the local disks in sync (with virtually no complexity)

There are some drawbacks to this environment:

• Complete CPU capacity is needed on both platforms for peak CPU capacity for active applications.

Applications can be run under Node-1 while Node-2 is always kept up to date, ready for failover of the storage as well as the applications which are sitting on that storage pool.

Dual-Node Shared Nothing Storage

Some people may not bee too impressed - there is still a node which is completely unused. This additional node may be considered a pure cost in an H-A or D-R environment. This is not necessarily true, if other strategies are taken into consideration.

For example, a dual-active node, where individual internal storage could be leveraged on dual active nodes through dual initiators, to completely leverage CPU capacity on both nodes during peak times.

The benefits are clear in this environment:

• External Storage is not required (with it's complexity and costs)


• Additional storage area network infrastructure is not required (with it's complexity and costs)


• The OS of the active node continually keeps all the local disks in sync (with virtually no complexity)


• 200% CPU capacity on two platforms can be leveraged during peak usage times

There are some drawbacks to this environment:

• Fail-over of a single node results in reduction to 100% of CPU capacity

Applications can be run under Node-1 and Node-2 while disks on the opposing node is always kept up to date, ready for failover of the storage as well as the applications which are sitting on that storage pool.

Multi-Node Shared Nothing Storage

The dual-active node share nothing architecture seems very beneficial, but what can be done in very typical three-tier environments?

Considering how simple it is to move around pools as well as zones, multi-node clustering can be done with a couple of simple scripts.

For example, a triple-active node, where individual internal storage could be leveraged on all three active nodes through triple initiators, to completely leverage CPU capacity on all nodes during peak times.


The benefits are clear in this environment:

• External Storage is not required (with it's complexity and costs)


• Additional storage area network infrastructure is not required (with it's complexity and costs)


• The OS of the active node continually keeps all the local disks in sync (with virtually no complexity)


• 300% CPU capacity across all platforms can be leveraged during peak processing times


• Failover of a single node means only a decrease to 200% CPU processing capacity

Applications can be run under Node-1, Node-2, and Node-3 while disks on the opposing nodes are always kept up to date, ready for failover of the storage as well as the applications which are sitting on that storage pool.

Application in Network Management

What does this have to do with Network Management?

Very often, there are multiple platforms which are used on polling platforms, with a high-availability requirement on an embedded database. There is usually a separate cost for H-A kits for applications as well as databases.

Placing each of the tiers within a Solaris Container is the first step to business optimization, higher availability, and cost reduction.


As a reminder, Oracle RDBMS can legally be run within a CPU Capped Solaris 10 Container, in order to reduce CPU licensing costs, leaving plenty of CPU available for failing over applications from other tiers. As additional capacity is needed by the business, the additional license can be purchased and the cap extended to other cores on the existing platform.

Pushing down the H-A requirements to the OS level eliminates application & license complexities and enables drag-and-drop load balancing or disaster-recovery under Solaris 10 or OpenSolaris using Solaris Containers. Running a RDBMS within a Capped Solaris 10 Container gives the business the flexibility to buy/stage hardware without having to pay the unused cpu cycles until they are actually needed.

- - - - - - - - - - - - - - - - - - -

Update - 2009-01-07: Another blog posting about this feature:

Solaris tip of the week: iscsi failover with COMSTAR


Update - 2019-10-21: Previous "Solaris tip of the week" no longer exists, transferred post:
https://jaydanielsen.wordpress.com/2009/12/10/solaris-tip-of-the-week-iscsi-failover-with-comstar/

I've been researching HA iscsi configurations recently, and I'd like to capture and share what I've learned about the COMSTAR stack. I have a simple demo that you can use for your own experiments...

 

More Work With ZFS

More Work With ZFS

The Last Time...

The last time ZFS was covered, an description of overall features were covered. How to use all of those features was uncovered. This post will try to cover some of the other features.

ZFS Sharing Overview

ZFS centralizes all directory sharing into a single command structure and removes the needs to manage arcane configuration files to deal with issue such as configuration, status, and persistency.

ZFS Sharing Protocols

The new ZFS suite offers protocol sharing of iSCSI, NFS as well as SMB(CIFS). There is a catch to this: the ZFS host must support a kernel implementations of the protocols - SMB(CIFS) is only supported under more recent releases of OpenSolaris and iSCSI is only supported under the Solaris families.

ZFS Sharing Stopping and Starting

ZFS uses a property to determine whether a filesystem mount is going to be shared or not.

To stop NFS sharing of a ZFS filesystem.

servera/root$ zfs set sharenfs=off u201

To start an NFS sharing of a ZFS filesystem.

servera/root$ zfs set sharenfs=on u201

Sharing Status

On the same server, one can check the "share" command to see what is being shared from all protocols, persistent or not. A listing of domestic sharing protocols that can be checked are in a configuration file on the sharing host

servera/admin$ cat /etc/dfs/fstypes
nfs NFS Utilities
autofs AUTOFS Utilities
cachefs CACHEFS Utilities

servera/admin$ share
- /u000 anon=60001,rw=servera "" - /u201 rw ""

On a foreign server, one can check to see what is being shared (via NFS protocol), persistent or not. A listing of foreign protocols that can be checked are in a configuration file on the remote host.

serverb/admin$ cat /etc/dfs/fstypes nfs NFS Utilities autofs AUTOFS Utilities cachefs CACHEFS Utilities

serverb/admin$ dfshares servera RESOURCE SERVER ACCESS TRANSPORT servera:/cdunix servera - - servera:/u201 servera - -

For the share and dfshares command, if no protocol is specified, then the "nfs" protocol is the default. A ZFS filesystem shared over NFS can be done using the "share" and "dfshares" command.

Sharing and Persistence

In most historic POSIX systems, there is a file referred to as "sharetab" (or some derivative of it) to review the sharing of filesystems. This is effective against any underlying filesystem (i.e. UFS, VxFS, ZFS, etc.) In the example below, cdunix is not on a ZFS filesystem.

servera/admin$ cat /etc/dfs/sharetab /u000 - nfs rw /u201 - nfs rw

If one is running a pure ZFS environment, persistence is held as a property. You can see the status of the ZFS file share through a ZFS command.

servera/admin$ zfs get sharenfs u201
NAME PROPERTY VALUE SOURCE
u201 sharenfs on local

Checking all shared protocols through ZFS is also possible, through parsing "all" properties option.

servera/admin$ zfs get all grep share
u201 sharenfs on default
u201 shareiscsi off default
u201 sharesmb off default

Checking the share status for all protocols from a foreign server is not as elegant. Individual protocols must be used, such as the "dfshares" command.