Meltdown - SPARC Solaris: The Only Safe Choice

Meltdown - SPARC Solaris: The Only Safe Choice

Abstract:

As the rest of the industry has been struggling with security vulnerabilities, SPARC Solaris platforms have been relatively quiet. Meltdown, otherwise known as CVE-2017-5754, has taken the world by storm. Operating Systems have long relied on Memory Management Units to isolate user application programs from the OS kernel. This had come to a screeching halt, leaving lesser secure systems in a world of hurt.

[Meltdown Logo, courtesy solaris.wtf]

Meltdown Vulnerability:

Some OS's will keep the Kernel Pages mapped into the same context as User Application Pages. This is often done for speed (i.e. linux) but places extra dependencies upon the MMU for isolation. Nearly all OS's had ceded this security concern to the CPU vendor, instead of applying the most secure practice in the OS architecture.

Meltdown:

As one vendor noted, SPARC Solaris is immune from Meltdown and about the only platform not subject to this critical vulnerability in the data center. This was accomplished by OS designers placing Kernel and User pages into different contexts, a design which added additional security, but at a performance cost that other OS designers in the industry were not willing to cede.

Conclusion:

Some Solaris systems, decades ago, may be affected, but nothing modern. Secure by Design is a typical decision for Solaris architects, a decision that has served them well for the decades they served a 64 bit OS to the user community, as other OS vendors played "catch up" in performance or features or functionality.

The Future - SPARC M8+ & Solaris

[SPARC Logo, Courtesy SPARC International]

The  Future - SPARC M8+ & Solaris

Abstract:

Solaris has been the heart of Sun Microsystems since 1982, with 32 bit SPARC RISC CPU since 1987. Fujitsu joined the SPARC/Solaris community in 1992, with others to follow. In 1995, SPARC went 64 bit, and has been ever since. In 2009, Oracle Corporation purchased Sun and all of it's SPARC & Solaris assets. Oracle & Fujitsu had been releasing processors & Solaris releases in rapid succession, ever since.

[SPARC Roadmap, courtesy Fujitsu]

Fujitsu SPARC Roadmap

In 2017, Fujitsu released one of the fastest processors on the market, which happened to be a SPARC. Since they, they had been very clear about it's roadmap for SPARC & Solaris, reaching out to a new SPARC release in 2020. Their roadmap has not changed in close to a year, so they appear to be on-track.

[SPARC & Solaris Roadmap, courtesy Oracle]

Oracle SPARC Roadmap

Shortly after Fujitsu's SPARC release, Oracle also released their SPARC M8 processor, which became the fastest socket in the world, once again. Also in September, Oracle release a roadmap where there was no future SPARC socket. This has been remedied in Spring 2018, where a snapshot of the official Oracle SPARC roadmap mirrors Fujitsu.

The spread between M6, M7, M8, and M8+ all seem to be spaced about 2.5 years apart, indicating not much of a change in the silicon release schedule from Oracle. Both Fujitsu & Oracle are now both using the same TSC fab for their SPARC silicon, which hints at a degree of consolidation outside of their roadmaps.

As with the way Embedded 10 Gig Ethernet was consolidated on the old T2+ socket, Network Management is wondering if Oracle will finally release the embedded Infiniband on the M8+ sockets. Embedded Infiniband was conspicuously missing from the "Sonoma" S7 release. (This would be a huge bonus to the Engineered Systems, to add a high degree of consolation & increase reliability & increased performance - TODAY.)

More interesting, there is a new hardware category called "Next Generation Storage", linked with the M8+ servers. Oracle had never been a true hardware company before purchasing Sun - they mostly re-sold hardware from third party vendors in their engineered systems, where Oracle felt they could add services value. Seeing SPARC in the roadmap, for engineered systems, is interesting, but will Solaris SPARC assume the role (with Solaris 11.4's new Storage features) or Intel Linux (with new Block volumes in Linux Storage Appliance)?

[Solaris Logo, Courtesy Sun Microsystems]

Oracle Solaris Roadmap

Oracle announced in early 2017 that SPARC & Solaris will move to an Agile Continuous Improvement cycle. Solaris 12 was erased from the roadmap, which was expected with Continuous Delivery, but created an odd amount of uncertainty in the media. Human resources alignments occurred in 2 batches, creating a large amount of uncertainty, but a new M8 processor was released. Assurances that Premier Support would continue to at least 2038!

January 2018, Solaris 10 went on Extended Support, meaning an uplift for patches are required, and 3 more years before patches cease. The final Premier Support patches were released, called 2018-01. The push to Solaris 11 is on!

Solaris 11.3 continued to get monthly SRU's (i.e. Solaris 11.3 SRU 30, Solaris 11.3 SRU 31, etc.) Open Beta of Solaris 11.4 was released, followed by a refresh, and finally an announcement that there will be a major point release of Solaris ever summer moving forward. The latest Solaris 11.4 refresh is nearly all 64 bit with buffer overflow protections, which is quit an accomplishment! The roadmap shows the next release, Solaris Next, which will likely be Solaris 11.5

Conclusions:

As the market uncertainty over SPARC & Solaris continues, new silicon and new operating system releases continue to occur. Long term industry players continue to release hardware. Solaris continues is march forward with aggressive new features, even pushing all components in the operating system to 64 bit. The only promise still outstanding is rebootless patching, possibly with KSplice. Wim, where is Solaris KSplice? You're in charge now, right?

State of The Art - SPARC M8 & Solaris

[SPARC M8 Socket, Courtesy Oracle Datasheet]

State of The Art - SPARC M8 & Solaris

Abstract:

The SPARC processor was developed by Sun Microsystems and had existed since the exit of major systems manufacturers from the Motorola 68K environment. Multiple manufacturers had always existed in this environment, to provide to consumers multiple supply chains in this commodity hardware market. The migration from 32 bit to 64 bit computing in SPARC occurred decades ago, as current computing systems still wrestle with the complexities. The next increment of hardware from Oracle was released September 17, 2017 - once again maintaining the fastest CPU sockets and SMP systems in the world. It is past 7 months since the release, it is long overdue for an article.

[SPARC M8 Block diagram, courtesy Oracle T8/M8 Architecture White Paper]

Block Diagram Layout

A logical block diagram can help the reader gain a better understanding of the heart of the system. That author of the Next Platform publication speculates, how four S7 processors may have been merged onto the same die or silicon, using a switched interconnect encapsulated in silicon.

Overall, it does appear to look like a fully switched quad-socket 32 core 256 thread SMP system, burned onto a single piece of silicon. This is a pretty astounding effort, any way one chooses to examine the diagram!

[SPARC M8 Processor with Oracle & Sun Microsystems SPARC family, Courtesy Oracle]

Architecture Changes

In general, processors have undergone many changes, mostly regarding consolidation of functions. As time progressed, the pattern of consolidating resulted in lower part counts, higher performance, higher reliability. Components were designed, sometimes combined, sometimes re-separated in various architectures:

• Integer Processing Units
• Floating Point Processing Units
• Memory Management Units
• Processor Socket Cross-Bar Switch
• 1st level cache
• 2nd level cache
• 3rd level cache
• Network Interfaces
• Video Processing
• Encryption Units
• De-encryption Units
• Decompression Units
• [DAX] Database Analytics Acceleration Units

The M8 is no exception... being a server & cloud oriented, video has been de-emphasized for some time & no longer developed actively by Oracle. Previous processors had integrated 10 Gig Ethernet (i.e. Sun Microsystems SPARC T2+) or attempted to integrate Infiniband (i.e. Oracle SPARC "Sonoma" S7) - but Oracle had decided to de-emphasize these commodity features. Oracle, being primarily a software company, concentrated on the most "bang for the buck" with faster cores, encryption, decryption, decompression, and database acceleration units.

[SPARC Solaris Layered Availability Infrastructure, courtesy Oracle SPARC T8/M8 RAS datasheet]

Socket Availability Architecture

The SPARC M8 Processor continues to play a vital role in the Availability Architecture of Solaris systems. With the innermost 4 layers of the onion, features like: parity, error correction, reissue on error in hardware, internal power management, voltage scaling, frequency scaling. Decades of design continue to exhibit robust features unavailable in commodity CPU architectures.

[SPARC Solaris DIMM Sparing Infrastructure, courtesy Oracle SPARC T8/M8 RAS datasheet]

Memory Availability Architecture

Normally, error correction exists on DIMM's chosen for the SPARC Servers. If one of the DIMM's repeatedly experiences errors on a SPARC M8, Solaris will initiate the process of retiring the faulty DIMM. The hardware can interleave memory across all 16 DIMM's, for maximum throughput, and can also facilitate interleaving memory requests across 15 DIMM's, in order to facilitate a DIMM retirement. During retirement, there is a slight decrease in the amount of available memory.

Virtualization Architecture

The SPARC M8 processor works in combination with OpenFirmware in order to implement the lowest levels of virtualization.

Physical Domains, which allows for electrically disconnected domains, is facilitated by the LOM (Lights Out Management) card, allowing for an OpenBoot instance to operate on each physical domain (note: this feature is only available on higher end systems.) There are no particular features within Physical Domains, which specifically benefit from the M8 processor.

Within the Physical Domain, a Hypervisor program is run in the OpenFirmware, allowing for virtual instances of OpenBoot to run for different groups of devices (i.e. console, memory, pci-cards slots, network cards, disk devces.) Different OS's may run inside of these logically separated OpenBoot instances. A Logical Domain can be live-migrated to a different Physical domain or even a different SPARC chassis, taking advantage of M8 enbedded engines at wire-speed such as Encryption, Decryption, and Decompression.

Zone virtualization is also available under Solaris, regardless of underlying architecture. The SPARC M8 brings hardware accelerated encryption, decryption, and uncompression for activities like live migration, similar to the benefits to Logical Domains.

Recent Processor Comparisons

The SPARC M8 processor has certainly advanced in performance over the past few generations of silicon, both from Oracle and Fujitsu. The Register was kind enough to put together a simple chart, illustrating the differences between each of the recent processors, each processor being the fastest socket of it's kind during it's release.
 

[Multivendor SPARC Comparison Chart, courtesy The Register]

Performance Comparisons

As The Register had pointed out, the new core's "wider instruction pipeline... coupled with the increase in clock frequency and the larger L1 code cache" sees to wring out the most recent performance gains from this silicon. These features help to define the 5th generation S5 core, located in the former [1/4 sized] S7 and now M8 processors. Boosting the clock rate nearly 25% from slightly over 4GHz M7 to an industry "unheard of" 5 GHz clearly is beneficial.

[SPARC M8 Security Features, courtesy Oracle Datasheet]

Security Comparisons

The Register also reported important enhancements in security, "it can perform hardware-accelerated cryptographic functions – such as AES, RSA and SHA-512 – at twice the speed of the M7." In addition, the SPARC M8 is the only processor in the industry which supports SHA-3 in silicon. With functions like Random Number Generators and [En|De]Crypto operations in hardware, there are no excuses for lack of security in a data center since there is virtually no cost or processing overhead. Crypto functions operate fast enough  to occur at wire-speed for network with virtually unnoticable CPU overhead, reserving the rest of the system resources for processing business/military data.

[SPARC M8 RDBMS Features, courtesy Oracle Datasheet]

Database & Analytics Comparisons

The massive thread count with 32x cores, 8x threads per core, 32x floating point units are not unusual for the SPARC M8, although these are already unusually large in comparison to other slower commodity & proprietary processors on the market. These significantly boost application & database performance to non-competitive proprietary Intel and other processors.

The SPARC M8 advances on the former "State of the Art" SPARC M7 with larger caches & faster processing for all workloads, including Database & Analytics.

The real secret of this processor lies in the Next-Generation Data Analytics Accelerator (DAX) engines. In-line decompression enables 200%+ increase in data held in memory for processing without performance impact at 120+ GB/second. Performance improvements of up to 700% for in-memory query acceleration is demonstrated! Oracle number processing is now managed through accelerators, short-cutting software routines to directly pass processing of numbers larger than 16 bytes to the hardware for all database workloads, resulting in 1000% increases in performance! Java 8 Streams API can be used to work with data-sets directly via the DAX, eliminating iteration operations [since they are implied] providing 2000% increase in performance on some operation types!

Conclusion

There are no competing processors in the same Integer or Floating Point Processing Class as the Oracle SPARC M8 processor. Full security, with nearly any cryptography library of choice, is no longer a burden with the SPARC M8... there are no security compromises with this processor, outrunning any workload on any similarly configured socket-to-socket configuration, even with full encryption enabled under SPARC. Once one tries to compare database processing or java application  processing, it is quickly observed that many industry players may be a decade or more away in performance of similar workloads. All of this, with the ability to virtualize, with no virtualization performance or monetary tax, makes this an exceptional core to any system in any data center. It will be difficult for any hardware vendor to build any kind of superior next generation processor in the first quarter of the 21st century... the SPARC M8 is the apex of modern day computing.

State of The Art - SPARC S7 & Solaris

[SPARC S7 Processor, Courtesy Oracle Data Sheet]

State of The Art - SPARC S7 & Solaris

Abstract:

The SPARC processor was developed by Sun Microsystems and had existed since the exit of major systems manufacturers from the Motorola 68K environment. Multiple manufacturers had always existed in this environment, to provide to consumers multiple supply chains in this commodity hardware market. The migration from 32 bit to 64 bit computing in SPARC occurred decades ago, as current computing systems still wrestle with the complexities. Oracle ceased producing horizontal scaling CPU sockets once purchasing Sun Microsystems. In June 2016, Oracle decided to re-enter the commodity market with the SPARC S7. NetMgt had published an article, but it was lost, and it was decided it was time to re-publish it again.

[Labeled Die Photo, courtesy Oracle Hot Chips 27 Presentation]

S7 "Sonoma" Floor Plan:

It can be clearly seen that the new die photo shows the use of 2x 4 Core Clusters. The cores are nearly the same 4th generation S4 cores, bundled in it's 32 core sister M7 processor. Glueless Coherency links were bundled, to scale an S7 system from 8 to 16 cores, with little external circuitry. With DDR4 memory interfaces on-chip, latency is cut down by eliminating external chips. Database Analytics Accelerators have been included, although not as many per-core as with the larger M7.

Close to 20% of the space used by an Infiniband network interface. The last time integrating network on-board silicon occurred was with the UltraSPARC T2+ (which integrated 10 Gig Ethernet) - but it was not released to customer facing production system.This was a significant disappointment to NetMgt, since an new Blade system with an Infiniband backplane allowing scalability to thousands of sockets would have been a welcome addition for cloud computing.

[Courtesy, Oracle's SPARC S7 Servers Technical Overview]

Architectural Changes:

The S7 is a smaller processor, returns to 8 cores on a die, similar to the T4 from back in 2011, but even outperforms processors from 2013 with higher cache, clock speed, and 4th generation core. It was designed to be a competitive socket (in price/performance) to commodity proprietary CPU's (instead of being the fastest performing socket in the market.)

[Dual-Die Photo, courtesy Oracle Hot Chips 27 Presentation]

Dual Socket Configuration:

The glueless dual-socket configuration allows for outstanding communication speeds between sockets. It is apparent that more bandwidth may be available over PCIe links than over the un-exposed Infiniband. Without the Infiniband exposed, the S7 looks more like an UltraSPARC IIIi.

[Infiniband Performance, courtesy Oracle Hot Chips 27 Presentation]

Unexposed Infiniband Performance:

The unexposed infiniband offered the possibility of significant packet performance improvement, over a PCIe card, even under high load. Note, the red Sonoma IB line mostly maintaining between 20-60 Millions of Packets per second.

A blade chassis connecting all minimal Sonoma "S7" blades (holding memory & 2 sockets) connecting back to infiniband storage in an MPP environment scaling across thousands of nodes could have been an amazing entry into Cloud Computing or Super Computing environments.

Conclusions:

While  NetMgt welcomes the new low-cost "Sonoma" S7 options, we see the form-factor produced as a "game changer" unrealized, by not placing the chip in a socket, exposing Infiniband, and into a chassis form factor which could most leverage it's strength. Such a socket could have easily replaced & out-performed the Intel based Storage subsystem Oracle's Infiniband native Engineered Systems. A such a socket in a blade chassis would have filled-out a gaping hole in Oracle's systems portfolio. Furthermore, the cost of memory is so high in the chassis that the cost difference between SPARC "Sonoma" S7 and SPARC M7 are marginal if selecting a chassis to perform LDom virtualization. It is a beautiful chip, for what it is, but a great opportunity lost.

HBA Firmware Update Required

HBA Firmware Update Required

Abstract:

When an OS update delivers firmware down to HBA cards, it is occasionally required to perform a manual link reset to enable the firmware This process describes it.

Symptoms:

When  rebooting Solaris 11, one may see the following WARNING messages to the console or /var/adm/messages log

Boot device: disk  File and args:
SunOS Release 5.11 Version 11.3 64-bit
Copyright (c) 1983, 2018, Oracle and/or its affiliates. All rights reserved.
WARNING: emlxs0: Firmware update required.
        (To trigger an update, a manual HBA or link reset using fcadm or emlxadm is required.)
WARNING: emlxs1: Firmware update required.
        (To trigger an update, a manual HBA or link reset using fcadm or emlxadm is required.)
WARNING: emlxs2: Firmware update required.
        (To trigger an update, a manual HBA or link reset using fcadm or emlxadm is required.)
WARNING: emlxs3: Firmware update required.
        (To trigger an update, a manual HBA or link reset using fcadm or emlxadm is required.)

Resolution:

Review the available devices to be reset
sun1801/root# fcinfo hba-port | grep "Device Name"
        OS Device Name: /dev/cfg/c5
        OS Device Name: /dev/cfg/c6
        OS Device Name: /dev/cfg/c10
        OS Device Name: /dev/cfg/c11

and perform the reset
sun1801/root# fcinfo hba-port | nawk '
 /Device Name/ { print "luxadm -e forcelip",$NF }' | ksh -x
+ luxadm -e forcelip /dev/cfg/c5
+ luxadm -e forcelip /dev/cfg/c6
+ luxadm -e forcelip /dev/cfg/c10
+ luxadm -e forcelip /dev/cfg/c11

Cisco c3550 Password Recovery from SPARC Solaris

Cisco c3550 Password Recovery from SPARC Solaris

Abstract:

Sometimes, old lab equipment may sit around, operating well for years without interruption. In some cases, the password for older network lab equipment may have had it's authentication passwords lost. Network equipment is often in the same rack as a remote server. A Solaris Server can be easily used to gain console on various serial based network equipment to perform password recovery.

SPARC Server:

Physical serial ports on SPARC Solaris servers are often defined as tty's, vs pts's (Pseudo Terminals.)
Log into a server via a pts or pseudo terminal:

Last login: Tue Feb 20 14:44:39 2018 from sun1593.daytono
Sun Microsystems Inc.   SunOS 5.9       Generic May 2002
INTR=Ctrl-C ERASE=Ctrl-H KILL=Ctrl-U
/dev/pts/1
sun9999/user$

Example of physical terminal ports available on servers with a hardware serial port:

sun9999/user$ ls -al /dev/tty* | head -4
lrwxrwxrwx 1 root other 26 Mar 12 2016 /dev/tty -> ../devices/pseudo/sy@0:tty
lrwxrwxrwx 1 root root 6 Mar 12 2016 /dev/ttya -> term/a
lrwxrwxrwx 1 root root 6 Mar 12 2016 /dev/ttyb -> term/b
lrwxrwxrwx 1 root root 30 Mar 12 2016 /dev/ttyp0 -> ../devices/pseudo/ptsl@0:ttyp0

The ttya port is used for console access, while ttyb can be used with a Cisco console cable. The default speed for older equipment is 9600 baud.

sun9999/user$ tip -9600 /dev/ttyb
connected

Process for password recovery or just normal console usage can begin.

Cisco Console Equipment:

Depending on the configuration, a single carriage return may result in a read-only prompt or an  authentication sequence.

Switch>

A normal power cycle of the switch may look like the following on the console.

Switch> Base ethernet MAC Address: 00:11:93:a1:70:80

Xmodem file system is available.

The password-recovery mechanism is enabled.

Initializing Flash...

flashfs[0]: 87 files, 4 directories

flashfs[0]: 0 orphaned files, 0 orphaned directories

flashfs[0]: Total bytes: 15998976

flashfs[0]: Bytes used: 6642176

flashfs[0]: Bytes available: 9356800

flashfs[0]: flashfs fsck took 16 seconds.

...done Initializing Flash.

Boot Sector Filesystem (bs:) installed, fsid: 3

Loading "flash:c3550-i9q3l2-mz.121-20.EA1a/c3550-i9q3l2-mz.121-20.EA1a.bin"...###########################################################################################################################################################################################################################################################################################################################################################################################################

File "flash:c3550-i9q3l2-mz.121-20.EA1a/c3550-i9q3l2-mz.121-20.EA1a.bin" uncompressed and installed, entry point: 0x3000

executing...

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is

subject to restrictions as set forth in subparagraph

(c) of the Commercial Computer Software - Restricted

Rights clause at FAR sec. 52.227-19 and subparagraph

(c) (1) (ii) of the Rights in Technical Data and Computer

Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.

           170 West Tasman Drive

           San Jose, California 95134-1706

Cisco Internetwork Operating System Software

IOS (tm) C3550 Software (C3550-I9Q3L2-M), Version 12.1(20)EA1a, RELEASE SOFTWARE (fc1)

Copyright (c) 1986-2004 by cisco Systems, Inc.

Compiled Mon 19-Apr-04 21:42 by yenanh

Image text-base: 0x00003000, data-base: 0x006B3454

Initializing flashfs...

flashfs[1]: 87 files, 4 directories

flashfs[1]: 0 orphaned files, 0 orphaned directories

flashfs[1]: Total bytes: 15998976

flashfs[1]: Bytes used: 6642176

flashfs[1]: Bytes available: 9356800

flashfs[1]: flashfs fsck took 8 seconds.

flashfs[1]: Initialization complete.

...done Initializing flashfs.

POST: CPU Buffer Tests : Begin

POST: CPU Buffer Tests : End, Status Passed

POST: CPU Interface Tests : Begin

POST: CPU Interface Tests : End, Status Passed

POST: Switch Core Tests : Begin

POST: Switch Core Tests : End, Status Passed

POST: CPU Interface 2nd Stage Tests : Begin

POST: CPU Interface 2nd Stage Tests : End, Status Passed

POST: CAM Subsystem Tests : Begin

POST: CAM Subsystem Tests : End, Status Passed

POST: Ethernet Controller Tests : Begin

POST: Ethernet Controller Tests : End, Status Passed

POST: ILP Controller Tests : Begin

POST: ILP Controller Tests : End, Status Passed

POST: Loopback Tests : Begin

POST: Loopback Tests : End, Status Passed

cisco WS-C3550-24-PWR (PowerPC) processor (revision H0) with 65526K/8192K bytes of memory.

Processor board ID CAT0829Z301

Last reset from warm-reset

Running Layer2/3 Switching Image

Ethernet-controller 1 has 12 Fast Ethernet/IEEE 802.3 interfaces

Ethernet-controller 2 has 12 Fast Ethernet/IEEE 802.3 interfaces

Ethernet-controller 3 has 1 Gigabit Ethernet/IEEE 802.3 interface

Ethernet-controller 4 has 1 Gigabit Ethernet/IEEE 802.3 interface

24 FastEthernet/IEEE 802.3 interface(s)

2 Gigabit Ethernet/IEEE 802.3 interface(s)

The password-recovery mechanism is enabled.

384K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address: 00:11:93:A1:70:80

Motherboard assembly number: 73-8100-08

Power supply part number: 341-0029-03

Motherboard serial number: CAT08290FL9

Power supply serial number: DTH082326P8

Model revision number: H0

Motherboard revision number: A0

Model number: WS-C3550-24PWR-SMI

System serial number: CAT0829Z301

Press RETURN to get started!

00:00:33: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan

00:00:35: %SYS-5-CONFIG_I: Configured from memory by console

00:00:36: %SYS-5-RESTART: System restarted --

Cisco Internetwork Operating System Software

IOS (tm) C3550 Software (C3550-I9Q3L2-M), Version 12.1(20)EA1a, RELEASE SOFTWARE (fc1)

Copyright (c) 1986-2004 by cisco Systems, Inc.

Compiled Mon 19-Apr-04 21:42 by yenanh

00:00:36: %SNMP-5-COLDSTART: SNMP agent on host Switch is undergoing a cold start

00:00:37: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down

00:00:38: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down

Switch>

A single carriage return would bring a prompt or authentication sequence.

Cisco Switch Password Recovery

On the 3550 model switch, there is a button on the front console, which can be depressed during the power down to power up transition, which will bring the switch into ROM mode.

Base ethernet MAC Address: 00:11:93:a1:70:80

Xmodem file system is available.

The password-recovery mechanism is enabled.

The system has been interrupted prior to initializing the

flash filesystem.  The following commands will initialize

the flash filesystem, and finish loading the operating

system software:

    flash_init

    boot

switch:

Note: The "System" light may flash 22 times, and cease flashing. Release the "mode" switch.


After the mode switch is released, the "System" light may continue to blink. Perform a "flash_init"

switch: flash_init

Initializing Flash...

flashfs[0]: 87 files, 4 directories

flashfs[0]: 0 orphaned files, 0 orphaned directories

flashfs[0]: Total bytes: 15998976

flashfs[0]: Bytes used: 6642176

flashfs[0]: Bytes available: 9356800

flashfs[0]: flashfs fsck took 16 seconds.

...done Initializing Flash.

Boot Sector Filesystem (bs:) installed, fsid: 3

switch:

Note: This may take about 12 seconds.


Perform "load_helper", rename the configuration file holding the password.

switch: dir flash:/

Directory of flash://

2    -rwx  0                        env_vars

3    -rwx  348                      system_env_vars

4    -rwx  5                        private-config.text

6    -rwx  616                      vlan.dat

8    drwx  192                      c3550-i9q3l2-mz.121-20.EA1a

7    -rwx  3940                     config.text

9356800 bytes available (6642176 bytes used)

switch: rename flash:config.text flash:config.old

switch: dir flash:/

Directory of flash://

2    -rwx  0                        env_vars

3    -rwx  348                      system_env_vars

4    -rwx  5                        private-config.text

6    -rwx  616                      vlan.dat

8    drwx  192                      c3550-i9q3l2-mz.121-20.EA1a

7    -rwx  3940                     config.old

9356800 bytes available (6642176 bytes used)

switch:

Note: When the switch reboots, the file will be regenerated.


Now, "boot" switch and "[control][c]" to gain user mode, if initial configuration is not needed.

switch: boot
Loading "flash:c3550-i9q3l2-mz.121-20.EA1a/c3550-i9q3l2-mz.121-20.EA1a.bin"...###########################################################################################################################################################################################################################################################################################################################################################################################################
File "flash:c3550-i9q3l2-mz.121-20.EA1a/c3550-i9q3l2-mz.121-20.EA1a.bin" uncompressed and installed, entry point: 0x3000
executing...
              Restricted Rights Legend
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706


Cisco Internetwork Operating System Software
IOS (tm) C3550 Software (C3550-I9Q3L2-M), Version 12.1(20)EA1a, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Mon 19-Apr-04 21:42 by yenanh
Image text-base: 0x00003000, data-base: 0x006B3454

Initializing flashfs...
flashfs[1]: 87 files, 4 directories
flashfs[1]: 0 orphaned files, 0 orphaned directories
flashfs[1]: Total bytes: 15998976
flashfs[1]: Bytes used: 6642176
flashfs[1]: Bytes available: 9356800
flashfs[1]: flashfs fsck took 8 seconds.
flashfs[1]: Initialization complete.
...done Initializing flashfs.
POST: CPU Buffer Tests : Begin
POST: CPU Buffer Tests : End, Status Passed
POST: CPU Interface Tests : Begin
POST: CPU Interface Tests : End, Status Passed
POST: Switch Core Tests : Begin
POST: Switch Core Tests : End, Status Passed
POST: CPU Interface 2nd Stage Tests : Begin
POST: CPU Interface 2nd Stage Tests : End, Status Passed
POST: CAM Subsystem Tests : Begin
POST: CAM Subsystem Tests : End, Status Passed
POST: Ethernet Controller Tests : Begin
POST: Ethernet Controller Tests : End, Status Passed
POST: ILP Controller Tests : Begin
POST: ILP Controller Tests : End, Status Passed
POST: Loopback Tests : Begin
POST: Loopback Tests : End, Status Passed
cisco WS-C3550-24-PWR (PowerPC) processor (revision H0) with 65526K/8192K bytes of memory.
Processor board ID CAT0829Z301
Last reset from warm-reset
Running Layer2/3 Switching Image
Ethernet-controller 1 has 12 Fast Ethernet/IEEE 802.3 interfaces
Ethernet-controller 2 has 12 Fast Ethernet/IEEE 802.3 interfaces
Ethernet-controller 3 has 1 Gigabit Ethernet/IEEE 802.3 interface
Ethernet-controller 4 has 1 Gigabit Ethernet/IEEE 802.3 interface
24 FastEthernet/IEEE 802.3 interface(s)
2 Gigabit Ethernet/IEEE 802.3 interface(s)
The password-recovery mechanism is enabled.
384K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:11:93:A1:70:80
Motherboard assembly number: 73-8100-08
Power supply part number: 341-0029-03
Motherboard serial number: CAT08290FL9
Power supply serial number: DTH082326P8
Model revision number: H0
Motherboard revision number: A0
Model number: WS-C3550-24PWR-SMI
System serial number: CAT0829Z301
         --- System Configuration Dialog ---
Would you like to enter the initial configuration dialog? [yes/no]:
00:00:33: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
00:00:37: %SYS-5-RESTART: System restarted --
Cisco Internetwork Operating System Software
IOS (tm) C3550 Software (C3550-I9Q3L2-M), Version 12.1(20)EA1a, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Mon 19-Apr-04 21:42 by yenanh
00:00:37: %SNMP-5-COLDSTART: SNMP agent on host Switch is undergoing a cold start

Press RETURN to get started!

00:01:29: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
00:01:30: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
Switch>

Note: the switch is now in unprivileged "user" mode.

Configuration work can be done from the "enable" privileged mode.

Switch> enable

Switch# show running-config

Building configuration...

Current configuration : 1995 bytes

!

version 12.1

no service pad

service timestamps debug uptime

service timestamps log uptime

no service password-encryption

!

hostname Switch

!

!

ip subnet-zero

!

!

spanning-tree mode pvst

spanning-tree extend system-id

!

!

!

!

interface FastEthernet0/1

 switchport mode dynamic desirable

!

interface FastEthernet0/2

 switchport mode dynamic desirable

!

interface FastEthernet0/3

 switchport mode dynamic desirable

!

interface FastEthernet0/4

 switchport mode dynamic desirable

!

interface FastEthernet0/5

 switchport mode dynamic desirable

!

interface FastEthernet0/6

 switchport mode dynamic desirable

!

interface FastEthernet0/7

 switchport mode dynamic desirable

!

interface FastEthernet0/8

 switchport mode dynamic desirable

!

interface FastEthernet0/9

 switchport mode dynamic desirable

!

interface FastEthernet0/10

 switchport mode dynamic desirable

!

interface FastEthernet0/11

 switchport mode dynamic desirable

!

interface FastEthernet0/12

 switchport mode dynamic desirable

!

interface FastEthernet0/13

 switchport mode dynamic desirable

!

interface FastEthernet0/14

 switchport mode dynamic desirable

!

interface FastEthernet0/15

 switchport mode dynamic desirable

!

interface FastEthernet0/16

 switchport mode dynamic desirable

!

interface FastEthernet0/17

 switchport mode dynamic desirable

!

interface FastEthernet0/18

 switchport mode dynamic desirable

!

interface FastEthernet0/19

 switchport mode dynamic desirable

!

interface FastEthernet0/20

 switchport mode dynamic desirable

!

interface FastEthernet0/21

 switchport mode dynamic desirable

!

interface FastEthernet0/22

 switchport mode dynamic desirable

!

interface FastEthernet0/23

 switchport mode dynamic desirable

!

interface FastEthernet0/24

 switchport mode dynamic desirable

!

interface GigabitEthernet0/1

 switchport mode dynamic desirable

!

interface GigabitEthernet0/2

 switchport mode dynamic desirable

!

interface Vlan1

 no ip address

 shutdown

!

ip classless

ip http server

!

!

line con 0

line vty 5 15

!

!

end

Switch#

The switch is ready to be configured.

A Simple Configuration

Name switch "C3550", encrypt "cisco" password in MD5, and save running configuration.

Switch# configure terminal

Enter configuration commands, one per line.  End with CNTL/Z.

Switch(config)# hostname C3550

C3550(config)# enable secret cisco

C3550# copy running-config startup-config

Destination filename [startup-config]?

Building configuration...

[OK]

C3550(config)# exit

Note: the simple "cisco" password above should not be used, choose a stronger password.


The saved password can  be tested via a restart of the switch.

Show Saved Startup Configuration

The running and startup configuration should be the same. Show the startup configuration.

C3550# show startup-config
Using 2041 out of 393216 bytes
!
version 12.1
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname C3550
!
enable secret 5 $1$nOpq$3Hg6AloyI74Vq0HV0uwTq0
!
ip subnet-zero
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
interface FastEthernet0/1
switchport mode dynamic desirable
!
interface FastEthernet0/2
switchport mode dynamic desirable
!
interface FastEthernet0/3
switchport mode dynamic desirable
!
interface FastEthernet0/4
switchport mode dynamic desirable
!
interface FastEthernet0/5
switchport mode dynamic desirable
!
interface FastEthernet0/6
switchport mode dynamic desirable
!
interface FastEthernet0/7
switchport mode dynamic desirable
!
interface FastEthernet0/8
switchport mode dynamic desirable
!
interface FastEthernet0/9
switchport mode dynamic desirable
!
interface FastEthernet0/10
switchport mode dynamic desirable
!
interface FastEthernet0/11
switchport mode dynamic desirable
!
interface FastEthernet0/12
switchport mode dynamic desirable
!
interface FastEthernet0/13
switchport mode dynamic desirable
!
interface FastEthernet0/14
switchport mode dynamic desirable
!
interface FastEthernet0/15
switchport mode dynamic desirable
!
interface FastEthernet0/16
switchport mode dynamic desirable
!
interface FastEthernet0/17
switchport mode dynamic desirable
!
interface FastEthernet0/18
switchport mode dynamic desirable
!
interface FastEthernet0/19
switchport mode dynamic desirable
!
interface FastEthernet0/20
switchport mode dynamic desirable
!
interface FastEthernet0/21
switchport mode dynamic desirable
!
interface FastEthernet0/22
switchport mode dynamic desirable
!
interface FastEthernet0/23
switchport mode dynamic desirable
!
interface FastEthernet0/24
switchport mode dynamic desirable
!
interface GigabitEthernet0/1
switchport mode dynamic desirable
!
interface GigabitEthernet0/2
switchport mode dynamic desirable
!
interface Vlan1
no ip address
shutdown
!
ip classless
ip http server
!
!
line con 0
line vty 5 15
!
!
end

C3550#

Note: the added configuration lines above are easily identified.

Test Startup Configuration  through Switch Reload

Reboot switch with "reload" to test simple configuration and privileged "enable" password.

C3550# reload
Proceed with reload? [confirm]
01:02:58: %SYS-5-RELOAD: Reload requested
Base ethernet MAC Address: 00:11:93:a1:70:80
Xmodem file system is available.
The password-recovery mechanism is enabled.
Initializing Flash...
flashfs[0]: 88 files, 4 directories
flashfs[0]: 0 orphaned files, 0 orphaned directories
flashfs[0]: Total bytes: 15998976
flashfs[0]: Bytes used: 6644224
flashfs[0]: Bytes available: 9354752
flashfs[0]: flashfs fsck took 16 seconds.
...done Initializing Flash.
Boot Sector Filesystem (bs:) installed, fsid: 3
Loading "flash:c3550-i9q3l2-mz.121-20.EA1a/c3550-i9q3l2-mz.121-20.EA1a.bin"...###########################################################################################################################################################################################################################################################################################################################################################################################################
File "flash:c3550-i9q3l2-mz.121-20.EA1a/c3550-i9q3l2-mz.121-20.EA1a.bin" uncompressed and installed, entry point: 0x3000
executing...

              Restricted Rights Legend
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706


Cisco Internetwork Operating System Software
IOS (tm) C3550 Software (C3550-I9Q3L2-M), Version 12.1(20)EA1a, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Mon 19-Apr-04 21:42 by yenanh
Image text-base: 0x00003000, data-base: 0x006B3454

Initializing flashfs...
flashfs[1]: 88 files, 4 directories
flashfs[1]: 0 orphaned files, 0 orphaned directories
flashfs[1]: Total bytes: 15998976
flashfs[1]: Bytes used: 6644224
flashfs[1]: Bytes available: 9354752
flashfs[1]: flashfs fsck took 8 seconds.
flashfs[1]: Initialization complete.
...done Initializing flashfs.
POST: CPU Buffer Tests : Begin
POST: CPU Buffer Tests : End, Status Passed
POST: CPU Interface Tests : Begin
POST: CPU Interface Tests : End, Status Passed
POST: Switch Core Tests : Begin
POST: Switch Core Tests : End, Status Passed
POST: CPU Interface 2nd Stage Tests : Begin
POST: CPU Interface 2nd Stage Tests : End, Status Passed
POST: CAM Subsystem Tests : Begin
POST: CAM Subsystem Tests : End, Status Passed
POST: Ethernet Controller Tests : Begin
POST: Ethernet Controller Tests : End, Status Passed
POST: ILP Controller Tests : Begin
POST: ILP Controller Tests : End, Status Passed
POST: Loopback Tests : Begin
POST: Loopback Tests : End, Status Passed
cisco WS-C3550-24-PWR (PowerPC) processor (revision H0) with 65526K/8192K bytes of memory.
Processor board ID CAT0829Z301
Last reset from warm-reset
Running Layer2/3 Switching Image
Ethernet-controller 1 has 12 Fast Ethernet/IEEE 802.3 interfaces
Ethernet-controller 2 has 12 Fast Ethernet/IEEE 802.3 interfaces
Ethernet-controller 3 has 1 Gigabit Ethernet/IEEE 802.3 interface
Ethernet-controller 4 has 1 Gigabit Ethernet/IEEE 802.3 interface
24 FastEthernet/IEEE 802.3 interface(s)
2 Gigabit Ethernet/IEEE 802.3 interface(s)
The password-recovery mechanism is enabled.
384K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:11:93:A1:70:80
Motherboard assembly number: 73-8100-08
Power supply part number: 341-0029-03
Motherboard serial number: CAT08290FL9
Power supply serial number: DTH082326P8
Model revision number: H0
Motherboard revision number: A0
Model number: WS-C3550-24PWR-SMI
System serial number: CAT0829Z301

Press RETURN to get started!

00:00:33: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
00:00:35: %SYS-5-CONFIG_I: Configured from memory by console
00:00:35: %SYS-5-RESTART: System restarted --
Cisco Internetwork Operating System Software
IOS (tm) C3550 Software (C3550-I9Q3L2-M), Version 12.1(20)EA1a, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Mon 19-Apr-04 21:42 by yenanh
00:00:35: %SNMP-5-COLDSTART: SNMP agent on host C3550 is undergoing a cold start
00:00:37: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
00:00:38: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
C3550>
C3550>enable
Password:
C3550#

Note: There is no remote access at this point and all interfaces are defaulted to VLAN1

The switch needs a complex password, but it will operate when  equipment is attached.