Showing posts with label dladm. Show all posts
Showing posts with label dladm. Show all posts

Thursday, December 29, 2011

Solaris 11: A Cloud in a Box!

Solaris 11: A Cloud in a Box!
Abstract:
Computing industry began with resource centralized on singularly large computing platforms. The microprocessor brought computing power into the hands of individuals in homes and offices, but information was still centralized in each location. The creation of The Internet allowed for the sharing of information between homes and offices, around the globe. Reliable server and telecommunications infrastructure was required to make it work, applications were somewhat limited to a handful of standard Internet protocols, such as HTTP. Cloud Computing has been coming of age over the past number of years, driving custom applications to proprietary API's, to move more applications into the Internet, but this is quickly changing as operating system vendors include more robust virtualization. Cloud Computing is really about the virtualization of Internet infrastructure, including servers, to a point where pieces do not have to reside on the internet, nor in an office, nor just a split between the two - but can reside anywhere, including entirely in a laptop. Solaris 11, the first Cloud Operating System, offers the ability to virtualize everything, from entire data centers across thousands of platforms, to thousands of platforms virtualized on a laptop.

Simulating The Cloud: A Practical Example

Joerg M., an Oracle employee and publisher of C0T0D0S0, discusses Solaris 11 with some of it's features, demonstrates the building of a cluster of virtual data centers within a single operating system instance. If someone runs a data center, they should consider reviewing the article to better comprehend the capabilities of what a "Cloud" could and should be.

It should be noted that "simnet" clause to the "create-simnet" and "modify-simnet" are formally undocumented, but documented in the OpenSolaris released source code, and leveraged in various other derived Open Source branches. One of the most important distributions being the Joyent SmartOS cloud operating system distributions.

Not Included, but Not Out Of Scope

What is not included in Joerg's example are actual systems on the edges of the cloud. Adding them is actually more trivial than adding the virtual routers which were created. Add virtual interfaces, virtual systems, databases to virtual systems, middleware to virtual systems, applications to virtual systems, add bandwidth & latency limitations to WAN links, add port limitations to virtual firewalls, etc.

Why Go Through the Exercize?

Once someone builds the entire datacenter "in the box", creation of the real data center becomes trivial. But why does this matter?
  • For the first time, real test environments can be simulated, soup-to-nuts, in an inexpensive way. There is no charge for virtualization in a Solaris world.
  • Costs can be reduced by placing all development systems into a couple of "clouds" for virtually any number (Solaris supports over 4000 zones on a single OS instance) of applications
  • Movement of an application from development to test is as easy cloning a Zone and instantiating the Zone on a Test platform.
  • Costs can be reduced by placing all test systems into a couple of clouds for virtually any number of applications
  • Deploying tested application is as easy as instantiating the cloned test Zone on a production system
  • Disaster recovery is as easy as instantiating the Zone on the dead physical system onto a physical system in an alternate data center.
  • Deploying production applications into a cloud is as easy as backing up the application and restoring it into the cloud - not to mention bringing it back.
  • The interactions of the application with Firewalls, WAN's and LAN's are all well understood, with everything being properly developed and tested, making each production deployment seamless
The effort, with a step-by-step process will ensure that there are no missed steps in the process to bringing virtualization to a business.

Implications to Network Management

The world is slowly exiting the physical world and Network Management is no longer about monitoring edge routers and links - it is about monitoring virtualized infrastructure. Orchestration is all about automated deployment and cloud providers are getting better at this. The missing piece to this puzzle is robust SNMP management of everything. The creation of network management infrastructure needs to happen in the development clouds first, then the test clouds, so when the jump to production is complete - the management infrastructure has already been simultaneously developed and tested, with the applications.

Tuesday, May 17, 2011

Sun V890: Forcing Full Duplex Under Solaris 10



Sun V890: Forcing Full Duplex Under Solaris 10

Abstract:

When a server boots, the network card will negotiate with the network switch how to communicate. Various settings such as speed and duplex are often part of the auto-negotiation protocol. Occasionally, a server will not negotiate with the network switch the highest possible throughput. The duplex setting may improperly negotiate to half, which could result in collisions and errors, but this can be resolved through manually setting the port characteristics.

Background:

The Sun V890 offers various ethernet ports out-of-the box when purchased from Sun. Some of these port types include: ge, eri, qfe. To see which ports are installed on a platform, one can use the "dladm show-dev" command.

Symptoms:

Bad negotiation is often determined by poor performance.
sunv890/user$ netstat -in
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
lo0 8232 127.0.0.0 127.0.0.1 8854 0 8854 0 0 0
qfe0 1500 192.127.224.0 192.127.224.33 6258 0 1460 0 0 0
qfe1 1500 172.39.0.0 172.39.5.33 73075033 0 1774172 0 0 0
qfe2 1500 192.127.254.0 192.127.254.9 10190603 0 6153115 176 546372 0
Determining Bad Negotiation:

If a network switch negotiates incorrectly, the results are clear with Solaris 10.
sunv890/root# dladm show-dev
ge0 link: down speed: 0 Mbps duplex: unknown
eri0 link: down speed: 0 Mbps duplex: unknown
qfe0 link: up speed: 100 Mbps duplex: full
qfe1 link: up speed: 100 Mbps duplex: full
qfe2 link: up speed: 100 Mbps duplex: half
qfe3 link: down speed: 0 Mbps duplex: unknown
Correcting Bad Negotiation:

The duplex can be fixed dynamically. To correct instance "2" of the "qfe" interface:
sunv890/root# ndd -set /dev/qfe instance 2
sunv890/root# ndd -set /dev/qfe adv_100T4_cap 0
sunv890/root# ndd -set /dev/qfe adv_100fdx_cap 1
sunv890/root# ndd -set /dev/qfe adv_100hdx_cap 0
sunv890/root# ndd -set /dev/qfe adv_10fdx_cap 0
sunv890/root# ndd -set /dev/qfe adv_10hdx_cap 0
sunv890/root# ndd -set /dev/qfe adv_autoneg_cap 0
Verifying Corrected Negotiation:

Duplex correction can be easily verified.
sunv890/root# dladm show-dev
ge0 link: down speed: 0 Mbps duplex: unknown
eri0 link: down speed: 0 Mbps duplex: unknown
qfe0 link: up speed: 100 Mbps duplex: full
qfe1 link: up speed: 100 Mbps duplex: full
qfe2 link: up speed: 100 Mbps duplex: full
qfe3 link: down speed: 0 Mbps duplex: unknown
Other Interfaces:

If the problem was experienced on instance "0" of the "eri" interface, the following commands could be used to dynamically adjust the interface to 100 Mbs and full duplex.
sunv890/root# ndd -set /dev/eri instance 0
sunv890/root# ndd -set /dev/eri adv_100T4_cap 0
sunv890/root# ndd -set /dev/eri adv_100fdx_cap 1
sunv890/root# ndd -set /dev/eri adv_100hdx_cap 0
sunv890/root# ndd -set /dev/eri adv_10fdx_cap 0
sunv890/root# ndd -set /dev/eri adv_10hdx_cap 0
sunv890/root# ndd -set /dev/eri adv_autoneg_cap 0
If the problem was experienced on instance "0" of the "ge" interface, the following commands could be used to dynamically adjust the interface to 1000 Mbs and full duplex with 802.3 transmit and receive flow control.
sunv890/root# ndd -set /dev/ge instance 0
sunv890/root# ndd -set /dev/ge adv_1000fdx_cap 1
sunv890/root# ndd -set /dev/ge adv_1000hdx_cap 0
sunv890/root# ndd -set /dev/ge adv_pauseTX 1
sunv890/root# ndd -set /dev/ge adv_pauseRX 1
sunv890/root# ndd -set /dev/ge adv_1000autoneg_cap 0




Kernel Inquiry:

The configuration and statistics regarding the network interfaces can also be retrieved through a kernel inquiry. This can be used for operating systems earlier than Solaris 10.
sunv890/root# kstat qfe
module: qfe instance: 0
name: qfe0 class: net

align_errors 0
allocbfail 0
babble 0
brdcstrcv 36739
brdcstxmt 493
buff 0
carrier_errors 0
...

Thursday, April 22, 2010

Solaris 10: Digging Into TCP/IP Problems


Solaris 10: Digging Into TCP/IP Problems

Abstract:
Transmission Control Protocol / Internet Protocol (TCP/IP) has been embedded into most UNIX platforms since the beginning days of the Internet. With the enhancement of TCP/IP protocols over the decades, configuration & debugging has become increasingly more sophisticated. In the process of converting to a 100% open source system, Solaris 10 has acquired newer tools to work through issues.

Key Files:
Some key files needed to validate a configuration:
/etc/inet/hosts
/etc/inet/ipnodes
/etc/inet/netmasks
/etc/inet/services
/etc/defaultrouter
/etc/hostname.{interface}
/etc/nsswitch.conf
/etc/resolv.conf
Key Commands:
Some key commands used to validate the configuration:
/usr/sbin/ifconfig
/usr/bin/netstat
/usr/sbin/dladm
/usr/sbin/inetadm
/usr/sbin/ping

Key Tasks:
Issues with debugging connectivity of a device or a service includes various steps.

Validate the machine host name is tied to an IP address:
sunt2000/root# grep `uname -n` /etc/inet/hosts /etc/inet/ipnodes
/etc/inet/hosts:192.168.254.7
sunt2000 loghost
/etc/inet/ipnodes:192.168.254.7 sunt2000 loghost
Validate the machine host name and ip address is tied to a network interface:
sunt2000/root# grep `uname -n` /etc/hostname.*
/etc/hostname.ipge2:
sunt2000
Validate a default gateway for traffic to pass off of the network:
sunt2000/root# grep -v ^# /etc/defaultrouter
192.168.254.2 1
Validate the configuration of the interface by the OS during the past reboot:
sunt2000/root# ifconfig ipge2
ipge2: flags=1000843 mtu 1500 index 4

inet 192.168.254.7 netmask ffffff00 broadcast 192.168.254.255

ether 0:14:4f:2:6a:5e
Validate the interface parameters for duplex and speed
sunt2000/root# dladm show-dev ipge2
ipge2 link: unknown speed: 100 Mbps duplex: full

Validate routing is set up from last reboot:
sunt2000/root# netstat -rn
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------- ------------- ----- ----- ---------- ---------
default 192.168.254.2 UG 1 6719031
192.127.254.0 192.168.254.7 U 1 23399 ipge2
224.0.0.0 192.168.254.7 U 1 0 ipge2
127.0.0.1 127.0.0.1 UH 1 26319 lo0
Validate traffic is passing without errors or collisions on the interface.
sunt2000/root# netstat -ni -I ipge2
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
ipge2 1500 192.168.254.0 192.168.254.7 419830401 0 538731765 0 0 0
Check to see what kind of name resolution is used on devices and services.
sunt2000/root# egrep '(^hosts|^services)' /etc/nsswitch.conf
hosts: dns files
services: files
When dns is being used, ensure the name services are running correctly.
sunt2000/root# nawk '/^nameserver/ { Cmd="nslookup www.oracle.com " $2 ; system(Cmd) }' /etc/resolv.conf
Server: 192.168.1.19

Address: 192.168.1.19#53

Non-authoritative answer:
www.oracle.com canonical name = www.oracle.com.edgesuite.net. www.oracle.com.edgesuite.net canonical name = a398.g.akamai.net.
Name: a398.g.akamai.net
Address: 63.216.54.107
Name: a398.g.akamai.net

Address: 63.216.54.106


Server: 192.168.24.231

Address: 192.168.24.231#53


Non-authoritative answer:

www.oracle.com canonical name = www.oracle.com.edgesuite.net. www.oracle.com.edgesuite.net canonical name = a398.g.akamai.net.
Name: a398.g.akamai.net

Address: 63.216.54.106
Name: a398.g.akamai.net
Address: 63.216.54.107
Check inet services which are not disabled.
sunt2000/root# inetadm | grep -v disabled
ENABLED STATE FMRI enabled online svc:/application/x11/xfs:default enabled online svc:/application/font/stfsloader:default enabled offline svc:/application/print/rfc1179:default enabled online svc:/network/rpc/smserver:default enabled online svc:/network/rpc/gss:default enabled online svc:/network/rpc/rstat:default enabled online svc:/network/security/ktkt_warn:default enabled online svc:/network/telnet:default enabled online svc:/network/nfs/rquota:default enabled online svc:/network/ftp:default enabled online svc:/network/login:rlogin enabled online svc:/network/shell:default enabled online svc:/network/rpc-100235_1/rpc_ticotsord:default enabled online svc:/network/bpcd/tcp:default enabled online svc:/network/vnetd/tcp:default enabled online svc:/network/vopied/tcp:default enabled online svc:/network/bpjava-msvc/tcp:default enabled online svc:/network/bootps/udp:default enabled online svc:/network/tftp/udp6:default enabled online svc:/network/rpc/cde-calendar-manager:default enabled online svc:/network/rpc/cde-ttdbserver:tcp
Check global properties for all inet services.
sunt2000/root# inetadm -p
NAME=VALUE
bind_addr=""
bind_fail_max=-1
bind_fail_interval=-1
max_con_rate=-1
max_copies=-1
con_rate_offline=-1
failrate_cnt=40
failrate_interval=60
inherit_env=TRUE
tcp_trace=FALSE
tcp_wrappers=FALSE
connection_backlog=10
List inet properties for any service which is suspect of not running correctly.
sunt2000/root# inetadm -l telnet
SCOPE NAME=VALUE
name="telnet"
endpoint_type="stream"
proto="tcp6"
isrpc=FALSE
wait=FALSE
exec="/usr/sbin/in.telnetd"
user="root"
default bind_addr=""
default bind_fail_max=-1
default bind_fail_interval=-1
default max_con_rate=-1
default max_copies=-1
default con_rate_offline=-1
default failrate_cnt=40
default failrate_interval=60
default inherit_env=TRUE
default tcp_trace=FALSE
default tcp_wrappers=FALSE
default connection_backlog=10
Conclusion:
For setting up and maintaining basic Network Management infrastructure under Solaris, these basic commands will take someone a long way.

Friday, April 16, 2010

Solaris 9: Missing dladm show-dev


Solaris 9: Missing dladm show-dev

Abstract:
Solaris 10 has included a new feature referred to as the Data Link Admin tool. This tool provides a simple way to configure and check the status of the layer 2 ethernet interfaces. Some of the information commonly used in dladm under Solaris 10 can be derived in Solaris 9.

Solaris 10: dladm show-dev
The Data Link Administration tool under Solaris 10 has some very nice features, including quickly seeing the interface name, speed, and duplex.

sunt2000# dladm show-dev
ipge0 link: unknown speed: 100 Mbps duplex: full
ipge1 link: unknown speed: 100 Mbps duplex: full
ipge2 link: unknown speed: 100 Mbps duplex: half
ipge3 link: unknown speed: 0 Mbps duplex: unknown


Solaris 9: kstat & nawk
A simple nawk script can be used on a Solaris 9 platform, to perform similar output.

sunt2000# kstat -p | nawk '/duplex/ || /speed/ { split($1,Array,":") ; Dev=Array[3] } /link_duplex/ && $2=="2" { Duplex[Dev]="full" } /link_duplex/ && $2=="1" { Duplex[Dev]="half" } /link_speed/ { if ( Duplex[Dev] == "" ) Duplex[Dev]="unknown" ; Speed[Dev]=$2 ; print Dev "\tlink: unknown\tspeed: " Speed[Dev] "\tMbit\tduplex: " Duplex[Dev] }'
ce0 link: unknown speed: 100 Mbit duplex: full
ce1 link: unknown speed: 1000 Mbit duplex: full
ce2 link: unknown speed: 1000 Mbit duplex: full
ce3 link: unknown speed: 1000 Mbit duplex: full
ce4 link: unknown speed: 0 Mbit duplex: unknown
ce5 link: unknown speed: 0 Mbit duplex: unknown

Monday, March 29, 2010

Administer Interfaces Under Solaris



Administer Interfaces Under Solaris

Abstract:


Under Solaris, users could configure some flat files and reboot a system in order to bring up their machine on an IP network. The startup scripts under Solaris traditionally leveraged the straight "ifconfig" command to add the network interfaces for the IP layer. The command for managing interfaces as the data link layer had been "dladm". Sun also bundled "IPFilter" into Solaris 10 for firewalling and NAT'ing. Configuring the various IP interfaces has been somewhat complex, so a new user friendly command has recently landed in OpenSolaris to help: "ipadm".

Configuration File Method:

The configuration files which are normally leveraged to configure a simple system before restart include:
  • /etc/inet/hosts
    Holds the IP Address & hostname of the machine. Can also be used for the default router. Example entries include:
    192.168.3.250 Ultra60 loghost
    192.168.3.1 WRT600 defaultrouter


  • /etc/inet/ipnodes
    Holds the IP Address & hostname of the machine. Can also be used for the default router. Example entries include:
    192.168.3.250 Ultra60 loghost
    192.168.3.1 WRT600 defaultrouter

  • /etc/netmasks
    Hosts the address of the network that the machine is sitting on. Example entry includes: 192.168.3.0 255.255.255.0

  • /etc/defaultrouter
    Holds the IP Address or host name of the default router. Example entry includes:
    defaultrouter

  • /etc/hostname.[interface]
    Holds the hostname for the physical[:logical] IP interface. Example for "ce0" includes:
    Ultra60
The "files" method is persistent across reboots, but of course requires a reboot. If a reboot is not possible, then follow the instructions below to perform the On-Line Method of configuration.

On-Line Method:

For people who can not afford to perform reboots of a system, interfaces can be added while the operating system remains on-line using the "ifconfig" command.
  • ifconfig [interface] plumb
    Puts the "plumbing" in place to configure an IP interface. Example command:
    ifconfig ce0 plumb

  • ifconfig [interface] addif [ipaddress/netmask]
    Assigns an address to a plumbed interface.
    ifconfig ce0 addif 192.168.3.2/24
In order for on-line changes to be persistent across reboots, it is important that the Configuration File Method has already been followed.



The New Method

With the PSARC/2010/080, OpenSolaris includes a unified interface to build interfaces on-line, which can also be made persistent.

For some examples on how to use the new OpenSolaris "ipadm" command are here.

We are all looking forward to this making it into Solaris!