VMWare Resolves Some Issues

VMWare 5.1 Resolves Some Issues

Abstract:
With the advent of simple and cost effective virtualization under Solaris 10, Zones, LDoms, and Virtual Box - pressure has been placed upon dominate virtualization vendors to create less expensive alternatives. VMWare, after being purchased by EMC, had decided to move in the opposite direction, making purchasing of VMWare very difficult, with odd pricing constraints in ESXi 5.0 in July 2011. The market has moved to 2012 and ESXi 5.1 has been released, fixing some of VMWare's problems.

Compatibility Issue Resolved:
If customers wanted to move an older VM to newer hardware, the VM's needed to be upgraded. In other words, there was compatibility issues which needed to be resolved. VM's created under ESX Server 3.5 and later will now run under ESXi 5.1 unchanged. This is good news for service providers.

No Longer Windows Bound:
Customers who had VMWare ESXi were required to use a lousy Microsoft Windows platform to manage the VMWare platform. When managing an ESXi server in a DMZ, this makes little sense for a service provider. This has now been resolved, with a web interface.

Memory Tax Issue Resolved:
The pricing constraints of ESXi 5.0 forced service providers to have to decide - is VMWare the correct hypervisor for the job... is Windows and/or Linux worth the aggravation of being nickel and dimed to death? When trying to determine hardware and hypervisor pricing for a new cluster where one does not know exactly how much memory will be required per instance because infrastructure is being purchased by a managed services provider before the first customer deal is sold, how does one know how much to buy?

Clearly, EMC's VMWare did not have a clue. The confusion that the pricing placed upon managed service providers negatively impacted purchasing of other EMC software products such as ITOI (aka Ionix, aka SMARTS) and RSA Archer, enVision, etc. If a managed service provider can not determine what to buy, they will not buy from that vendor. Solaris is clearly the better choice for Network Management, and other vendors are clearly the better choice for tools bound to VMWare & Windows.

The removing of the memory constraints for ESXi 5.1 was a good move, to simplify pricing. EMC Software is now in a better position to compete against other virtualized platforms.

Outstanding Core Issues:
For reasonable flexibility in the data center environment, when there is a spike in usage, there needs to be a way to easily migrate heavy usage live instances to lower utilized hypervisors. Dynamic migration with autobalancing is included with Oracle LDom's, but not quite there yet with VMWare.

When dealing with network virtualization, if one is trying to emulate a WAN environment, one could spin up dozens of zones under a Solaris 11 platform, and apply the WAN characteristics to the virtual network (latency, throughput, etc.) Technology like Solaris Crossbow is missing from VMWare.

Conclusions:
VMWare is a great benefit to the Windows and Linux world, but constraints by the vendor made purchasing difficult and implementation less desirable. Some of the issues have been resolved, but management is not yet what it needs to be for managed service providers.

Network Management at EMC World 2012

 

[EMC World 2012 Man - courtesy: computerworld]

Network Management at EMC World 2012

Abstract:
EMC purchase network management vendor SMARTS with their InCharge suite, a number of years ago, rebranding the suite as Ionix. EMC purchased Voyence, rebranding it as NCM (Network Configuration Manager). After EMC World 2012, they completed the acquisition of Watch4Net APG (Advanced Performance Grapher.) The suite of these platforms is now being rolled into a single new brand called EMC IT Operations Intelligence. EMC World 2012 was poised to advertize the new branding in a significant way.

Result:
EMC World 2012 in Las Vegas, Nevada was unfortunately pretty uneventful for service providers. Why was it uneventful?

The labs for EMC IT Operations Intelligence did not function. There were a lot of other labs, which functioned, but not the Network Management labs. EMC World 2012 was a sure "shot-in-the-head" for demonstrating, to service providers, the benefits of running EMC Network Management tools in a VM.

After 7 days, EMC could not get their IT Operations Intelligence Network Management Suite running in a VMWare VM.

Background:
Small customers may host their network management tools in a VMWare VM. Enterprises will occasionally implement their network management systems on smaller systems, where they know they will get deterministic behavior from the underlying platform.

Service Providers traditionally run their mission critical network management systems on larger UNIX Systems, so as to provide instant scalability (swap in CPU boards) and 99.999 availability (reboot once-a-year, whether they need to or not.)

The platform of choice in the Service Provider market for scalable Network Management platforms has been SPARC Solaris, for decades... clearly, for a reason. This was demonstrated well at EMC World 2012.

The Problem:
Why not host a network management platform in a VMWare infrastructure? Besides, the fact that EMC could not make it happen, after 1 year of preparation, and 7 days of struggling... there are basic logistics.

Network Management is dependent upon ICMP and SNMP.  Both of these protocols are "connectionless protocols" - sometimes referred to as "unreliable protocols". Why would a network management platform use "unreliable protocols"?

The IETF understands that network management should always be light (each poll is a single packet, while a TCP protocol requires a 3-way handshake to start the transaction, poll the single packet, then break down with another 3-way handshake. Imagine doing this for thousands of devices every x seconds - not very light-weight, not very smart. A "connection based protocol" will also hide the nature of an unreliable underlying network, which is what a network management platform is supposed to expose - so it can be fixed.

Now stick a network management platform in a VM, where the network connection from the VM (holding an operating system, with a TCP/IP stack), going down through the hypervisor (which is another operating system, with another TCP/IP stack, which is also sharing the resources of that VM with other VM's.) If there is the slightest glitch in the VM or the hypervisor, which may cause the the packets to be queued or dropped - the actual VMWare infrastructure will signal to the Network Management Centers that there is a network problem, in their customer's network!

Politics:
Clearly, someone at EMC does not understand Network Management, nor do they understand Managed Service Providers.

The Network Management Platform MUST BE ROCK SOLID, so the Network Operations Center personnel will NEVER mistake a alerts in their console from a customer's managed device as a local performance issue in their VM.

With EMC using Solaris to reach into the Telco Data Centers,  EMC later using Cisco to reach into the Telco Data Centers - EMC is done using their partners. VMWare was the platform of choice, to [not] demonstrate their Network Management tools on. Cisco was the [soon to be replaced] platform of choice, since EMC announced they will start building their own servers.

Either someone at EMC is sleeping-at-the-wheel or they need to get a spine to support their customers. Either way, this does not bode well for EMC as a provider of software solutions for service providers.

 

Business Requirements:
In order for a real service provider to reliably run a real network management system in a virtualized environment:

• The virtualized platform must not insert any overhead.
• All resources provided must be deterministic.
• Patches are installed while the system is live.
• Engagement of patches must be deterministic.
• Patch engagement must be fast.
• Rollback of patches must be deterministic.
• Patch rollback must be fast.
• Availability must be 99.999.  

Solutions:
There are many platforms which fulfill these basic business requirements, but none of them are VMWare. Ironically, only SPARC Solaris platform is currently supported by EMC for IT Operations Intelligence, EMC does not support SPARC Solaris under VMWare, and EMC chose not to demonstrate their Network Management suite under a platform which meets service provider requirements.

Today, Zones is about the only virtualized technology which offers 0%-overhead virtualizataion. (Actually, on SMP systems, virtualizing via Zones can increase application throughput, if Zones are partitioned by CPU board.) Zones, to work in this environment, seem to work best with external storage providers, like EMC.

Any platform which offers 0% virtualization penalty with ZFS support can easily meet service providers technical platform business requirements. Of these, the top 3 are probably the best supported by commercial interests

• Oracle SPARC Solaris
• Oracle Intel Solaris
• Joyent SMART OS
• OpenIndiana
• Illumian
• BeleniX
• SchilliX
• StormOS

Conclusion:
Today's market is becoming more proprietary each passing day. The movement towards supporting applications only under proprietary solutions (such as VMWare) has demonstrated it's risk during EMC World 2012. A network management provider would not be well advised to use any network management tool which is bound to a single proprietary platform element and does not support POSIX platforms.

Windows 8: Getting More ZFS'ish

Windows 8: Getting More ZFS'ish

Abstract:
Storage has always been a part of operating systems. Over time, storage capabilities have increasingly became more sophisticated in operating systems, consuming features from 3rd party partners. Occasionally, a vendor will release very sophisticated increments to their operating systems. Windows 8 is projected to receive more ZFS-like features, to make it more competitive with Solaris, when Sun release ZFS over a half-dozen years ago.


History:
Intel based systems had relied upon a file system referred to as FAT since the late 1970's. Systems during this time period occasionally offered only upper case characters, FAT was case insensitive. During the 1980's, FAT16 was released, to deal with larger hard drives. The 1990's saw the introduction of FAT32, to deal with longer file names. These file system were, for the most part, primitive - targeting floppy disks and hard disks, identifying physical storage through a letter of the alphabet.

Microsoft, the key Intel based operating system vendor, promised an Object Oriented file system during the 1990's and a Relations Database filesystem during the 2000's. Both of these projects were canceled, with an interim file system called NTFS, to drag users out of the proverbial stone ages. NTFS was too proprietary bound to the Microsoft operating system) and too heavy to bring to portable storage devices, like camera flash cards and USB flash sticks, which still carry the baggage of the ancient FAT file system.

Most UNIX systems included file system management bound to individual disks during the late 60's and into the early 1970's, but they quickly moved to a virtual file system, allowing disks to be added (mounted) anywhere in a single tree, transparent to the applications. During the 1980's, Sun Microsystems extended this abstraction to virtualize the file system to network based devices, as well as physical devices. Third-Party vendors, like Veritas, created Logical Volume Management systems with their own third-party file systems, to consolidate multiple physical disks into a single virtualized physical device.

Sun Microsystems introduced in 2005 the first major advancement in file system technology, in decades. Gone was the forcing of users to use separate drive letters of DOS and NTFS to desgnate physical devices; gone was separate volume management system to virtualized multiple physical devices into a seeing device; gone was separate file system layer to sit on top of a virtual or physical device. ZFS was created to take operating systems into the next decade, with aspects consolidating a hybrid 64-128 bit internal structures.

Windows 8 Storage Spaces:
Microsoft quietly announced new features in a Windows 8 Beta via a blog post. The Register published a short article regarding Storage Spaces.

Microsoft will introduce in Windows 8 what it calls
Storage Spaces – a method of putting drives into a
virtual pool from which self-healing virtual disks
can be created, with some resemblance to ZFS features.

Details of these virtual disks – the aforementioned
Storage Spaces – were described in a 4,400-word
deep-dive blog poston Thursday, introduced by
Microsoft Windows Division head, Steven Sinofsky,
and written by a member of Redmond's Storage
and File System team, Rajeev Nagar.

How robust will the new system be, if it is released in Windows 8 Beta?

Storage Spaces is somewhat like ZFS, although it has no
deduplication and lacks other ZFS features. However, it is
a start – and Microsoft will probably add features such as
snapshots, replication, deduplication, and, maybe, compression.

The Register posted an article where Open Source OpenSolaris based Nexenta discusses Windows 8 Storage Spaces.

Windows 8 Storage Spaces papers over some previous
shortcomings, but does not solve a limit on the total
TB that a file system can store, which is 16TB. It
still lacks double or triple parity RAID, which means
your data is at risk since the odds of two failures
on a RAID group, and data loss, increases with large
drives that take more time to rebuild. There is also
no ability to snapshot and replicate your data.
Finally, there is no end-to-end data integrity -
there’s nothing like the cryptographic-strength
256 bit checksums of ZFS-based solutions like NexentaStor.

Analysis:
Will Microsoft's new Storage Spaces be able to displace NTFS or even FAT, or will Storage Spaces find itself in the company of former Microsoft Object or Relational file systems? This question will be ultimately answered in 10 more years.

Microsoft has a long history of trying to innovate and failing in storage. Microsoft has a long way to catch up. With virtually unlimited resources derived from a PC monopoly and constant inbound royalties from ancient FAT - they can not be counted out.

Network Management:
What will be the impact on Network Management? Nothing, if Storage Spaces follows other storage advancements.

Will Microsoft implement it's storage technologies with management extensions directly to SNMP, to make it truely network friendly?

Microsoft has the possibility to take leadership away from new ZFS owner, Oracle, on the Internet, if true SNMP management is taken into consideration. If not, Microsoft will be left a half-dozen year in the rears, continuing to play catch-up with the market leader.

Solaris Zones: Rapid Application Deployment

Solaris Zones: Rapid Application Development

Abstract:
Many development systems are based upon dedicated hardware infrastructure, which limits the number of simultaneous development releases. With Solaris Zones, multiple application releases can be developed at the same time on a platform, where the developed application is encapsulated, to be moved to test, and rapidly cloned into production.

Description:
Solaris Zones offer the following features:

• isolation
• rapid boot
• built-in rollback
• system mobility
• cloning
  

Presentation
A use-case by a telephone carrier is illustrated in the following Solaris Zone presentation.
[html] Solaris Zones Dev, Test, and Deployment Presenation

Additional Information
[html] Solaris Zones, Containers, Resource Management Guide
[html] Solaris Zones Developer Guide

Solaris Tab Update: Solaris 11 & Crossbow

Solaris Tab Update: Solaris 11 & Crossbow

New resources have been added to the Solaris Tab, primarily concerning Solaris 11.

There is a helpful PDF document demonstrating Virtual Networking via "Crossbow".

Network Management Connection

Setting up a completely virtualized server and switch environment on a single platform meets various requrements such as: portable network management demonstrations, framework to build network management test labs, and a framework to simulate and test network management applications in a WAN environment, without purchasing the hardware.

Powerful frameworks like Crossbow are available under Solaris 11 derivative operating systems like Solaris 11 Express, OpenSolaris, OpenIndiana, Illumos, etc.

Solaris Reference Material
2010-12 [PDF] Set Up a Virtual Network Automatically with Solaris 11 Express
2011-03 [PDF] Solaris 11 ISV Adoption Guide
2011-06 [HTML] Lab: Introduction to the Solaris ZFS File System
2011-06 [HTML] Lab: Protecting Your Applications with Solaris 10 Security
2011-06 [HTML] Lab: Protecting Your Applications with Solaris 11 Security
2011-06 [HTML] Lab: Installing Solaris 11 Express in Oracle VM VirtualBox

SNMP Page Update: Solaris 10 LDoms / Oracle VM Server for SPARC

SNMP Page Update: Solaris 10 LDoms / Oracle VM Server for SPARC

The Network Management SNMP page has been updated, adding a reference to Solaris 10 LDoms, or more recently called Oracle VM Server for SPARC.

There is an SNMP management infrastructure for the SPARC "T" series, which can be leveraged (free of cost) to provide multiple domain management fault and performance management. Capabilities include: reviewing the cpu/memory/disk/network/virtual-network resources, oberving logical domain stops/starts, and even stopping/starting logical domains through SNMP.

Why does this sound so foreign?

Because no one else does it for free, that is why... just another reason why Network Management resources are familiar & skilled with SPARC and Solaris.

SNMP - Solaris 10 Management Interface Base

LDoms 2.1 - [RFC] [MIB] [HTML] - Using the Oracle VM Server for SPARC Management Information Base Software

Cloud Computing: Amazon, Joyent, etc.

Cloud Computing: Amazon, Joyent, etc.

Abstract:
Cloud computing is no panacea - it is just a single tool in the tool belt of the Infrastructure Architect. There are concerns and options for customers to be concerned about.



One Vendor Example:
For people depending upon Cloud Computing, it should be noted that one of the most famous vendors, Amazon, even experiences outages. On 2011 April 21, there was a 10 hour outage. Amazon referred to it as a "network outage", but some data was irreconcilably lost.



Another Vendor:
Other vendors are experienced in the cloud computing model. Joyent has been providing cloud computing expertise since 2004. Joyent provides superior disk, cpu, and memory performance over Amazon. Amazon is clearly not the only game in town.


Programming Models:
Joyent offers varied cloud platform models, but perhaps one of their most interesting model is based upon Node.js, where event driven server side programming can be done with Javascript in a cloud as a service.

Conclusion:
Don't use customer facing applications, which should require H-A capability, solely in the cloud. If you have a customer facing application that is mission critical, ensure your capability is replicated to another provider or keep primary or redundant resources under your own control. When choosing a vendor, keep in mind that there are many options to meet most business requirements.

Joyent: Sun Out of the Clouds

Joyent: Sun Out of the Clouds

Joyent is offering a special to Solaris users:

• Open Solaris based SmartMachines

Open Solaris SmartMachines outperform Amazon EC2 on basic metrics:

• CPU is 5x faster
• Disc I/O is 14x faster
• Memory I/O 3x faster

Build high performance web applications with a 1GB SmartMachine for just $45 per month (special price offered on a new machine April 15th, 2011 and will remain in effect until the machine is decomissioned.

After the offer expires, the price will be a normal cost of $80 per month higher.)

Solaris Crossbow Virtual Wire: Network in a Box

Solaris Crossbow Virtual Wire: Network in a Box

Abstract:

For 8 years, Sun has been re-developing the TCP/IP stack under Solaris. Nicolas Droux is involved as one of the core architects in Solaris in the process of re-architecting the TCP/IP stack. At the 23'rd Large Installation System Administration Conference (LISA-09), Nicolas presented over a short session describing the new features in Solaris TCP/IP from Project Crossbow.

Problems

• Host Virtualilzation
  
• Service Virtualization
  

Key Issues to Solve

• Virtualizing Hardware NIC's
• Zones Sharing a NIC
• Maintain Performance
• The desire is to allow the virtualized network stack to use as much of the hardware as possible.
• Allow the Virtual Machines to understand how much bandwidth they are allowed to use, to keep zones from stepping on one another.
• Management integrated into the stack itself, to avoid users having to look at multiple man pages.
• Security to ensure badly behaved applications are not injecting bad packets on a shared network

8 Years of Development

• Old code based upon Steams of solutions to resolve
  
• closer integration of IP to TCP layers
• data link, mac to IP
• new interface to device drivers (Project Nemo)
• IP QoS integrated, simplified, and made more efficient
  
• Crossbow integrated at MAC layer
• Requested more modern NIC features from hardware more hardware rings buffers, DMA Channels, and rich classifiers... building new features into the TCP/IP stack
  

Enablers & Key Opportunities

• Server and Network Consolidation
  
• Open Networking
• Cloud Computing
  

Features

• Hardware Lanes, to assign traffic to virtual NIC's, buffers, kernel threads, interrupts, the CPU threads, Zones, and/or Virtual Machines!
• Stack adjusts flow based upon server load or traffic load, with ability to adjust interrupts, so large chains of packets can be pulled from the NIC without an interrupt per packet penalty
• Virtual NIC's, pseudo-MAC instances, can be configured with bandwidth, priorities, and link aggregation, and assign V-NIC's on top
• Bind: VLAN and Priority Flow Control to a V-NIC; hardware lan to a Switch
  
• Virtual switch built automatically whenever 2 VNIC's are assigned to a Data Link
• Virtual Switch can be built on EtherStubs, isolated from real hardware
• Assigning a CPU Pool to a VNIC is coming

Implications to Hardware

• Zones can replace real machines in a model in a Solaris model on a laptop
  
• Virtual Switches can replace real switches in a Solaris model on a laptop
  
• Virtual Routers can replace real routers in a Solaris model on a laptop
• The configuration can be deployed in a production data center

Implications to Services: Crossbow Flows

• Flows describes a type of traffic moving through a network
• Flows can be described by: Services, Transport, Port Number, etc.
  
• Properties can be attached to flows: Bandwidth, CPU, Priorities, etc.
  
• Flows can be created on NIC's an V-NIC's
  

Question & Answers

• Bandwidth can be assigned to a NIC, Bandwidth Guarantees to allow bursting was on the roadmap in 2009.
  

OpenSolaris 2009.06 - Network Virtualization

OpenSolaris 2009.06 - Network Virtualization

Network Virtualization Technology: Project Crossbow

Sun has been working at re-architecting the TCP/IP stack in Solaris for Virtualization for close to 3 years, making progress each year with new features. OpenSolaris 2009.06 exhibits some of the most recent enhancements
http://link.brightcove.com/services/player/bcpid1640183659?bctid=24579687001

Network infrastructure in Solaris has been re-written at the NIC, Driver, and Socket levels - all the way up the stack.

Network Virtualization has to do with dedicated resources and isolation of network resources. They are talking about multiple: Hardware Ring Buffers in a NIC, TCP/IP Stacks in a Kernel, Kernel Ring Buffers in a Stack.

http://www.opensolaris.com/use/ProjectCrossbow.pdf
"Crossbow is designed as a fully parallelized network stack structure. If you think of a physical network link as a road, then Crossbow allows dividing that road into multiple lanes. Each lane represents a flow of packets, and the flows are architected to be independent of each other — no common queues, no common threads, no common locks, no common counters."

Some of the more interesting results of this integration: create networks with no physical NIC cards; create switches in software; assign bandwidth to a virtual NIC card (vNIC); assign CPU resources to a vNIC; assign quality of service (QoS) attributes to a vNIC; throttling protocols on a vNIC; virtualize dumb NIC's via the kernel to look like smart NIC's; switch automatically between interrupt and polled modes.

The implications are staggering:

• Heavy consumption of network resources by an application does not necessarily have to step-on other mission critical applications running in another virtual server
• Priorities for latency sensitive protocols (ex. VoIP) can be specified for traffic based upon various packet policies, like Source IP, Destination IP, MAC address, Port, or Protocol
• Security is enhanced since Solaris 10 containers no longer have to share IP stacks for the same physical NIC, but physical NIC's can now have multiple IP stacks for each container
• Multiple physical ports can be aggregated into a single virtual port and then re-subdivided into multiple virtual NIC's so many applications or many virtual servers can experience load sharing and redundancy in a simplified way (once at the lowest layer instead of multiple times, for each virtual machine)
  
• Older systems can be retained for D-R or H-A since their dumb NIC's would be virtualized in the kernel and the newer NIC's with newer equipment can be added into the application cluster for enhanced performance
• Heavily used protocols will switch a stack into "polled mode" to remove the overhead of interrupts to the overall operating system, providing better overall system performance, as well as providing faster network throughput to competing operating systems
• Enhanced performance at a lower system resource expense is achieved by tuning the vNIC's to more closely match the clients mean flow control can happen at the hardware or NIC card level (instead of forcing the flow control higher in the TCP stack)
• Modeling of applications and their performance can be done completely on a laptop, all application tiers, including H-A, without ever leaving the laptop - allowing architects to test the system performance implications by making live configuration settings
• Repelling DoS attacks at the NIC card - if there is a DoS attack against a virtual server's vNIC card, the other virtual servers do not necessarily have to be impacted on the main system due to isolation and resource management, and packets are dropped at the hardware layer instead of at the kernel or application, where high levels of interrupts are soaking up all available CPU capacity.

Usually, adding & leveraging features like QoS and Virtualization will decrease performance to an operating system, but with OpenSolaris, adding these feature with a substantial re-write of code, enabled a substantial increase in read & write throughput over Solaris as well as substantial increase in read throughput (with close to on-par write throughput) in comparison to Linux on the same hardware.
http://www.opensolaris.com/learn/features/networking/networkperformance/

This OpenSolaris technology is truly ground-breaking for the industry.

Usage of Network Virtualization in Network Managment

In the realm of Network Management, there is usually a mix of unreliable protocols (ICMP and UDP) with reliable protocols (TCP sockets.) The unreliable protocols are used to gather (ICMP, SNMP) or collect (Syslog) data from the edge devices while reliable protocols are used to aggregate that data within the management platform cluster.

While the UDP packets are sent/received, they can be dropped under times of high utilization (event storms, denial of service attacks, managed network outages, etc.) - so applying higher quality of service to these protocols becomes desirable to ensure the network management tools have the most accurate view of the state of the network.

Communication to internal system, which are aggregating that data, require this data for longer term usage (i.e. monthly reporting) and must be maintained (i.e. backups) - but these subsystems are no where near as important to maintaining an accurate state of the managed network when debugging an outage, which affects the bottom line of the company. These packets can be delayed a few microseconds to ensure the critical packets are being processed.

Enhanced performance in the overall TCP/IP stack also means more devices can be managed by the network management platform while maintaining the same hardware.

Implementation of Network Virtualization in Network Management

The H-A platform can be loaded up with OpenSolaris 2009.06 and the LDOM holding the Network Management application can be live-migrated seamlessly in minutes.
http://blogs.sun.com/weber/entry/logical_domain_mobility_between_solaris

After running on the production H-A platform for a time, the production platform can be upgraded, and the LDOM migrated back in minutes.

Conclusion

Operating systems like OpenSolaris 2009.06 offer to the Network Management Architect new options in lengthening asset lifespan, increasing return-on-investment for hardware assets, ensuring better system performance of network management assets, ensuring the best possible network management team performance possible.