How to Kill a Zombie in Solaris

How to Kill a Zombie in Solaris

Abstract:

When a parent spans a child process, the child process will return a signal to the parent once the child process has died or was terminated. If the parent dies first, the init process inherits the children, and will receive the signals once the children die. This process is called "reaping". Sometimes, things do not go as planned. It is a good topic for Halloween.

[artwork for "ZombieLoad" malware, courtesy zombieloadattack]

When things do not go as planned:

It may take a few minutes for the exit signal to be reaped by a parent or init process, which is quite normal.

If children processes are dying and the parent is not reaping the signals, the child remains in the process table and becomes a Zombie, not taking Memory or CPU, but consuming a process slot. Under modern OS's, like Solaris, the process table can hold millions of entries, but zombies still consumes kernel resources and userland resources when process tables need to be parsed.

Identifying Zombies

Zombies are most easily identified as "defunct" processes.

# ps -ef | grep defunct
root 1260 1 0 - ? 0:00 

This defunct process would normally be managed by the parent process, which is "1" or init, but in this case we can clearly see that this process is not disappearing.

# ps -ef | grep init
root 1 0 0 Oct 25 ? 8:51 /sbin/init

But why call them Zombies and not just Defunct?

$ ps -elf | egrep '(UID|defunct)'
 F S  UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
 0 Z root 125 4549 0 0   -  -    0  -     -     ?   0:00

The "S" or "State" flag identifies the defunct process with a "Z" for Zombie, and all can see them.

(Plus, this is being published on Halloween, or All Hallows' Eve, the day before All Hallow's Day or All Saints' Day... this is when people remember the death of the "hallows" or Saints & Martyrs, who had passed on before. So, let's also remember the deaths of the processes!)

[The Grim Reaper, courtesy Encyclopedia Britannica]

To Kill a Zombie:

How does one kill a Zombie?
Well, they are already dead... in the movies, they are shot in the head.
In the modern operating system world of Solaris, we seek the reaper, we Don't Fear The Reaper.

The tool is called Process Reap or "preap" - the manual page is wonderfully descriptive!

# preap 1260
1260: exited with status 0

It should be noted, processes being traced can not be reaped, damage can occur to the parent process if the child is forcibly reaped, and the OS may also put restrictions on reaping recently terminated processes.

To force a reaping, one can place a proverbial "bullet in the head" of the zombie.

# preap -F 125
125: exited with status 0

So, there we go, two dead zombies, see how they no longer run.

Conclusion:

This administrator had personally seen poorly written C code, leaving thousands of zombies behind daily. The application  development team no longer had no C programmers on their staff, so this was a good option. It should be carefully exercised on a development or test box, to evaluate the results on the application, before conducing a procedure in production.