Check Point Firewall – Nokia IPSO CST Hanging or Taking Forever?

BlueCoatIs your Nokia IPSO CST Not finishing?
Trying to run a CST on your Nokia, but seems to be taking forever?

I have had the same issue on various Nokia security appliances running on IPSO 4.2 and older. The problem ended up being hung process that was spawned by the CST program. It seems that CST calls “fw tab -u -t”, and sometimes it just gets hung up, but will look to the user like the whole CST process is just hung. Here is a sample so you can get a visual:

FIREWALL123[admin]# cst
IPSO 3.7x or later kernel found...
CST version 8-19-04
 
======================= N O T I C E:  VOYAGER LOCKS =======================
On IPSO 3.8.1, please make sure you are logged out of Voyager. CST gathers certain 
information from clish, which may not work when there is a configuration lock in place 
established by an active Voyager session.
========================= E N D   O F   N O T I C E    =========================
 
Continue? [y] y
IPSO-4.2-BUILD029 detected...
Generating IPv4 configuration summary...done
Generating IPv6 configuration summary...done
Gathering cpu utilization data...done
Gathering memory utilization data...done
Gathering interface statistics...done
Gathering standard ipsoinfo data...done
Gathering additional ipso information...
   vrrp data...done
   route data...done
   rip data...done
   ospf data...done
   bgp data...done
   dvmrp data...done
   pim data...done
...done
Gathering system logs...ls: cst-FIREWALL123-01.26.2012-1938/logs/messages*: No such file or directory
...done
Processsing vmcore files...none on this system.....done
Gathering firewall data... Invalid flavour 'general' for product 'vpn'. Use 'cpstat' without any arguments to see supported products and flavours.
 Invalid flavour 'fwz' for product 'vpn'. Use 'cpstat' without any arguments to see supported products and flavours.
...done
Fetching cpinfo... 
cpinfo (I:0110):        Beginning ...
cpinfo (I:0116):        Latest cpinfo build: http://www.checkpoint.com/techsupport/downloadsng/utilities.html#cpinfo

—THIS IS WHERE THE CST PROCESS NORMALLY HANGS—

The CST process is performing many operations, and collecting all kinds of data. One piece of data it grabs is some CP tables with “fw tab”, which by checking on it with “ps aux”, will show that it just keeps running due to an undisclosed error within “fw tab”, causing CST operation to hang. So we need to find the PID of “fw tab” (looking for the long-running one) and use the “kill” command to send a signal 15 (SIGTERM) that will terminate the “fw tab” process and allow the CST process to proceed, and finish. So go ahead and open another SSH connection to the firewall, and run a “ps auxw”. The resulting output of “ps auxw” will be similar to the sample below:

FIREWALL123[admin]# ps auxw
USER       PID %CPU %MEM   VSZ  RSS    TT  STAT STARTED       TIME COMMAND
root     26287  0.0  0.0   472  228    p1  R+   8:40PM     0:00.01 ps -auxw
root     20046  0.0  0.0     0    0    ??  Z    -          0:00.00  (clishd)
root     14583  0.0  0.0     0    0    ??  Z    -          0:00.00  (clishd)
root     14584  0.0  0.0     0    0    ??  Z    -          0:00.00  (clishd)

---TRUNCATED---

root     26065  0.0  0.2  2572  1816   p3  I+    7:41PM    0:00.05 /opt/CPinfo-10/bin/cpinfo -o cst-FIREWALL123-01.26.2012-1939/cpinfo.v
root     26232  0.0  4.9  45492 51056  p3  I+    7:42PM    0:07.78 /opt/CPsuite-R65/fw1/bin/fw tab -u -f
root     26274  0.0  0.7  4092  7376   p0  Ss+   8:40PM    0:00.17 /bin/clish
root     26281  0.0  0.2  3016  2140   ??  S     8:40PM    0:00.01 /bin/clishd serving 26274
root     26282  0.0  0.0   496  372    p1  Ss    8:40PM    0:00.02 -/bin/csh
root     12735  0.0  0.0     0    0    ??  Z     -         0:00.00  (clishd)
FIREWALL123[admin]# kill -15 26232

To kill the PID of the long running service, identify the PID number, and execute the “kill PID” command. In our case, we issued “kill 26232”. Once you have killed the PID of “fw tab”, flip back over to your first SSH session, and in a minute or so, the CST process will be finishing up – again refer to the sample:

cpinfo (I:0112):        Embedding files ...
cpinfo (I:0120):        Output file  - cst-FIREWALL123-01.26.2012-1938/cpinfo.FIREWALL123.01.26.2012-1938
cpinfo (I:0111):        Done
done
Creating index...done
Creating archive file...done
Searching for system and process core files...done
Fetching routing daemon state...done
Checking for CST update......none found.
done
Compressing final archive...done
Output saved to cst-FIREWALL123-01.26.2012-1938.tar.gz
FIREWALL123[admin]#

Great, now you should have your CST completed!

ps auxw |grep “fw tab -u -f”