EMC Ionix: Architecture and Integration Basics
Abstract:
Network Management platforms perform monitoring, auditing, and management work of computing infrastructure. Most network management platforms target a particular aspect of management: Fault, Performance, or Configuration. SMARTS produced a fault managegment product suite called InCharge, which was later purchased by EMC and branded as Ionix - based upon the phrase "keep your eye on it". Integration into EMC Ionix is straight forward, leveraging a couple of basic command.
Architecture:
The Ionix infrastructure is based upon a publish-subscribe system. Individual Managers (i.e. Availability Manager [AM], MPLS Manager, etc.) perform polling of devices and publish the results, Adapters (SNMP Trap, Syslog, etc.) perform simple gathering of information from foreign systems, Open Integration [OI] consolidates information from multiple adapters and publishes the information, and a Manager of Managers called Service Assurance Manager [SAM] subscribes to information from them all. A broker tracks all components.
Integration Points:
The Managers, Open Integration, and Service Assurance Manager can be integrated to via the following commands:
- dmctl
Performs individual queries and updates to a manager or manager-of-managers - sm_adapter
Subscribes or publishes to a manager or manager-of-manager
The sm_adapter a native mechanism to perform advanced interfacing through the proprietary internal language called "asl" scripting.
The "asl" scripting is out of scope of this article.
DMCTL:
The DMCTL interface offers the following options:
SparcSolaris/User$ dmctlTo attach to a manager, like a SAM:
Domain Manager Control Program (V7.2.0.1) -- Type 'help' for a list of commands.
dmctl> help
Commands:
attach [domain]
clear [class::instance::event]
create [class]::[instance]
consistencyUpdate
correlate
delete [class]::[instance]
detach
execute [program] [[arg1] ...]
findInstances [class-regexp]::[instance-regexp]
get [class]::[instance][::[property]]
getClasses
getEvents [class]
getEventDescription [class]::[event]
getInstances [[class]]
getModels
getOperations [class]
getPrograms
getProperties [class]
getThreads
insert [class]::[instance]::[property] [value]
invoke [class]::[instance] [op] [[arg1] ...]
loadModel [model]
loadProgram [program]
notify [class::instance::event]
ping
put [class]::[instance]::[property] [value1] [[value2] ...]
quit
remove [class]::[instance]::[property] [value]
restore [file]
shutdown
status
save [file] [[class]]
SparcSolaris/User$ dmctl
Domain Manager Control Program (V7.2.0.1) -- Type 'help' for a list of commands.
dmctl> attach SAM-03
Server SAM-03 User: admin
admin's Password: XXXXXXXXXX
Attached to 'SAM-03'
To retrieve basic notification instances from a SAM:
dmctl> getInstances ICS_NotificationNote, the above example, the underscore "_" is the field separator. The underscore is escaped using double underscores. The retrieved instance is formatted with the following characteristics:
NOTIFICATION-Host_ABC__ABLD25__BR_Down
NOTIFICATION-Host_ABC__ACAQ04__ID_Down
NOTIFICATION-Host_ABC__ACAQ07__ID_Down
NOTIFICATION-Host_ABC__ACAQ08__ID_Down
NOTIFICATION-Host_ABC__ACBC01__BR_Down
Notification-{Device-Class}_{Device-Host-Name}_{Event} This was a simple event notification. The device could be extended with an additional set of flas to uniquely define a managed resource, but this is beyond the scope of this article.To subscribe to a live stream of events from a SAM using dmctl:
SparcSolaris/User$ dmctl -s SAM-03 subscribe .*::.*::.*All the properties of an event can be retrieved via dmctl:
Server SAM-03 User: admin
admin's Password: XXXXXXXXXX
1297883020 Wed Feb 16 14:03:40 2011 NOTIFY ICS_Notification::NOTIFICATION-Host_ABC__ACDB05__ID_Down::RootNotification 1.00
1297880934 Wed Feb 16 13:28:54 2011 NOTIFY ICS_Notification::NOTIFICATION-Host_ABC__ANVR02__BR_Down::RootNotification 1.00
1297880633 Wed Feb 16 13:23:53 2011 NOTIFY ICS_Notification::NOTIFICATION-Host_ABC__ANND02__ID_Down::RootNotification 1.00
1297880934 Wed Feb 16 13:28:54 2011 NOTIFY ICS_Notification::NOTIFICATION-Host_ABC__ANHS04__ID_Down::RootNotification 1.00
SparcSolaris/User$ dmctl -s SAM-03 get ICS_Notification::NOTIFICATION-Host_ABC__ACDB05__ID_DownSubscribing to an Open Integration manager is also possible:
Server SAM-03 User: admin
admin's Password: XXXXXXXXXX
Properties of ICS_Notification::NOTIFICATION-Host_ABC__ACDB05__ID_Down:
Acknowledged = FALSE
AcknowledgmentTime = 0
Active = TRUE
AggregatedBy = { }
Aggregates = { }
AuditTrail = {
{
22
1297883024
SYSTEM
Action completed successfully...
Remedy-AutoOpen-Ticket
}
...
SparcSolaris/User$ echo "" | dmctl -s OI-30 subscribe .*::.*::.*Subscribing to an Open Integration manager for a complete details is also possible with some nawk glue:
1297891133 Wed Feb 16 16:18:53 2011 NOTIFY ICS_Notification::NOTIFICATION-Host_ABC__ACSH02__BR_Down::RootNotification 1.00
1297891133 Wed Feb 16 16:18:53 2011 NOTIFY ICS_Notification::NOTIFICATION-Host_ABC__ACBI06__BR_Down::RootNotification 1.00
1297891133 Wed Feb 16 16:18:53 2011 NOTIFY ICS_Notification::NOTIFICATION-Host_ABC__ANMZ03__BR_Down::RootNotification 1.00
SparcSolaris/User$ echo "" | dmctl -s OI-30 subscribe .*::.*::.* |SM_ADAPTER
nawk 'NF==8 || NF==9 { gsub("::"," ") ; print "get " $8 "::" $9 }' |
dmctl -s OI-30
Domain Manager Control Program (V7.2.0.1) -- Type 'help' for a list of commands.
Attached to 'OI-30'
dmctl>
Properties of ICS_Notification::NOTIFICATION-Host_HDB__ACSH02__BR1_Down:
Acknowledged = FALSE
AcknowledgmentTime = 0
Active = TRUE
AggregatedBy = { }
Aggregates = { }
AuditTrail = {
...
The sm_adapter command has variety of options:
SparcSolaris/User$ sm_adapter --help
Usage: sm_adapter [options...] [[rule-set]]
Arguments:
* [rule-set] ASL rules file.
Options:
--broker=[location] Alternate Broker location as host:port.
Also -b [location].
--model=[model] Name of model library to load. Also -M [model].
--dynamic Load dynamic model files.
--name=[name] Start a server registered under [name].
Also -n [name].
--port=[port] Alternate registration port. Use with --name.
--timeout=[secs] Set the timeout for server interaction. The
timeout applies to the back-end connection
except when using the subscriber front end, in
which case it applies to the front end. The
argument is in seconds, and can be a decimal
value. If the --timeout option appears with no
value, 600 seconds is used. By default, there
is no timeout.
--wait Wait for initial driver to complete.
Rule-Set Options:
-D[var]=[value] Override value for a rule set variable.
--verify Validate rules only.
Front-End Options:
--file=[path] Read input from a file. Also -f [path].
--tail=[path] Read input by tailing a file from the current
position. Also -t [path].
--tailFromStart=[path] Read input by tailing a file from the beginning.
--program=[cmd] Read input from a command pipeline. Also -p [cmd].
--field-separator=[c] Translate 'C' to the field separator (FS) marker.
Valid only in conjunction with --file, --tail or
--program. Also -F [c].
--subscribe=[sub] Use the subscriber front-end. Subscriptions
are sent to the server specified with the
--server option. The [sub] parameter is the
subscription request.
If [sub] is 'topology' a subscription for
topology changes is requested.
If [sub] is of the form '[name]/n' then
a subscription to NL [name] is requested.
Note that only one NL subscription may be
specified.
If [sub] is of the form
C::I::E[/paev], 'C', 'I', 'E' are regexp
patterns representing the classes, instances,
and events to which to subscribe. The letters
following a slash (/) are subscription qualifiers:
'p' means subscribe to problems; 'a' means
subscribe to aggregates (impacts); and 'e' means
subscribe to events. If none of these are
present, 'p' is assumed. 'v' means run in
verbose mode, which turns on subscription
control messages.
Otherwise, [sub] is a profile name; that profile
specifies what subscriptions are to be requested.
A profile name may optionally be followed by the
/v qualifier.
Multiple --subscribe options can be specified.
--subscribeProp=[sub] Subscribe to property changes.
[sub] is of the form C::I::P[/v], 'C', 'I', 'P'
are regexp patterns representing the classes,
instances,and properties to which to subscribe.
The patterns are optionally followed by the /v
qualifier which, turns on the subscription
of the control messages too.
Multiple --subscribeProp options can be specified.
--smoothing=[num] Event smoothing interval. This parameter is
used by the subscriber front-end to smooth
event notifications (and clears) received
from the server. Only events (or clears) that
stay active (or cleared) for [num] seconds
are fed into the input stream. [num] must be a
non-negative integer. The default value is 0
which disables smoothing.
--ignoreOld Ignore old notifications. This parameter is
used by the subscriber front-end. Notifications
for events that were active at the before this
adapter connected are not fed to the input
stream.
Back-End (Server) Options:
[--server=self] Connect driver to local repository; the
default.
--server=null Do not connect to any server. Useful for
debugging offline in combination with
--traceServer.
--server=[name] Connect driver to remote server.
Also -s [name].
--rserver=[name] Auto-reconnect driver to remote server.
Also -S [name].
--description=[desc] Description of this adapter;
sent to remote server.
--mcast=[name] Connect driver to a local subscription server.
Trace Options:
--traceRules Trace rule compilation.
--traceServer Trace interactions with the back-end server.
--traceParse Trace rule matching.
--trace Enable all tracing. Also -d.
Standard Options:
--help Print help and exit.
--version Print program version and exit.
--daemon Run process as a daemon.
--logname=[name] Use [name] to identify sender in the system log.
Default: The program's name.
--loglevel=[level] Minimum system logging level. Default: Error.
--errlevel=[level] Minimum error printing level. Default: Warning.
--tracelevel=[level] Minimum stack trace level. Default: Fatal.
[level]: One of None, Emergency, Alert,
Critical, Error, Warning, Notice, Informational,
or Debug. Fatal is a synonym for Critical.
--facility=[facility] Non-Windows only. A case-insensitive string which
identifies the facility to use for syslog messages.
[facility]: One of Cron, Daemon, Kern, Local0-Local7,
Lpr, Mail, News, Uucp, User. Default: Daemon.
--output[=[file]] Redirect server output (stdout and stderr). The
file name is [file], or the --logname value if
[file] is omitted. Log files are always placed
in $SM_LOGFILES or $SM_WRITEABLE/logs.
--accept=[host-list] Accept connections only from hosts on
[host-list], a comma-separated list of host
names and IP addresses. --accept=any allows
any host to connect. Default: --accept=any.
--useif=[ip-address>] Use this IP address as the source/destination
interface address for SNMP and ICMP packets.
-- Stop scanning for options.
For more information:
file:/opt/InCharge7/SAM/smarts/doc/html/usage/index.html
http://www.EMC.com/
A notification list can be defined in an OI or a SAM for individual gui's to subscribe to. Notification lists can also be subscribed to via the sm_adapter:
SparcSolaris/User$ echo "" | sm_adapter -s OI-30 --subscribe='ALL_NOTIFICATIONS/n'
1295993287|CONNECT|OI-30|
1295993287|NL_CHANGE|Host|ABC_ACDI69_BR|Down|ID_2C?aPBE|
1295959726|NL_NOTIFY|Host|ABC_ANKP07_ID|Down|ID_lO(G:BE|
1295988119|NL_NOTIFY|Host|ABC_ACBG04_ID|Down|ID_2C?aPBE|
Note, with the sm_adapter output, the information can be parsed using the vertical pipe "|".
The sm_adapter can run individual "asl" script to perform the parsing in real time, but that is beyond the scope of this article.
Conclusion:
Integrating into Managed Service Provider frameworks for Network Management such as EMC Ionix is fairly straight forward and can be done by competent staff with POSIX scripting capabilities.
