Sun, 19 Mar 2006

Service Scripts
That connectivity script has several of the features of what I think of as 'service grade system monitor scripting'. These are long running scripts that continually monitor the status of some system characteristic, and do something useful with the results. The simpler scripts just record their results to a log file. More complex scripts can also scan results for alarm conditions and fire alarms as appropriate. (There's always an element of reinventing snmp management software functionality when treading this path. But there can be good reason to roll your own.)
These service grade scripts:

• check for status while running in a continual loop.
• on startup, will not launch a second loop if they're already running (to facilitate relaunch from the OS task scheduler.)
• log results to a length limited output file.
• can open a channel to the resource in question, typically via the filesystem, telnet, http, ftp or snmp.
• aren't particulary dependent on where they sit in the filesystem, or on the current working directory.
• tend to provide both positive and negative feedback.
• may detect alarm conditions and send alarms as appropriate (say, via email or sms).

Whenever we get surprised by a system outage or anomaly, we consider ways to automate the handling of its next occurance. In the case of the outage yesterday, it took us a while to pinpoint the breakage to the specific edge connection, and once we did, we didn't have a precice profile trace of the outage. If that particular breakage happens again, we'll consider adding a watcher to the new logfile to fire alarm messages as appropriate.

posted at: 14:15 | path: | permanent link to this entry

Connection Logger Script
here's that logger script I made yesterday.

@goto start
pixPasswordHere




enable
pixEnablePasswordHere
ping 55.111.22.3
exit

exit

(the above cruft is used by nc.exe as telnet session input)
(replace passwords and neighbor IP address as appropriate.)

:: expected output:
10:54a Sun -- 55.111.22.3 response received -- 10ms
10:55a Sun -- 55.111.22.3 response received -- 10ms
10:56a Sun -- 55.111.22.3 NO response received -- 1000ms 
10:56a Sun -- 55.111.22.3 response received -- 10ms
10:57a Sun -- 55.111.22.3 response received -- 10ms


:start
@echo off

:: -----------------
:: pingNN.cmd
::
:: keep a running log of connection state from edge PIX to upstream interface
:: (periodically telnet into the PIX and save response of ping to its neighbor)
::
:: uses: 
::   netcat (aka, nc.exe) for scripting the telnet connection.
::   perl
::   unique.pl perl script to filter out contiguous duplicate lines of text.
::   limitAt.bat keeps a file from growing past the specified size
:: -----------------


set PixIP=10.10.2.2

set delaySeconds=20
set loopDelay=perl -e "sleep %delaySeconds%"

set maxLogfileSize=5000000

set mylogdir=c:\limited
if not exist %mylogdir% md %mylogdir%
set mylogfile=%mylogdir%\pingNN-fromPIX.log

:: check age of %mylogfile%
if not exist %mylogfile% goto checked
for /f %%a in ('perl -e "use File::stat;print time-stat('%mylogfile%')->mtime" ') do set age=%%a
:: quit if already running
if %age% LSS 300 (
    echo Already appears to be Running...
    goto :eof
    ) else (
  echo.>>%mylogfile%
  )

:checked

:loop
for /f %%a in ('time/t') do (
  for /f %%x in ('date/t') do (
    for /f "tokens=*" %%m in ('nc -i 5 %PixIP% ^<%~fs0 ^| find "response" ^|perl c:\bat\unique.pl') do (
      echo %%a %%x -- %%m
      echo %%a %%x -- %%m >> %mylogfile%
      )
    )
  )
%loopDelay%
if exist %mylogfile% call c:\bat\limitAt %maxLogfileSize% %mylogfile% old
goto loop

Outage this morning

Got a call from Erik yesterday morning. The network (methownet.com) was experiencing intermittent outages. Erik had already done most of the troubleshooting. The only additional info needed was specifically where the connectivity broke down. After a bit more poking around, we learned that the break seemed to happen between the device (firewall) on the outer edge of our network and our providers device on that same subnet.

I called our provider to let them know that we were needing some attention, and then wrote a script to keep an ongoing log of the connectivity status for that critical link.

posted at: 08:11 | path: | permanent link to this entry