Installing Linux to a headless appliance

We have four x86-platform “server” appliances left over from an abandoned VoIP project by a consulting company that is now no longer in business. What to do with them? Install Linux, of course, and deploy them as remote network probes for a Smokeping network.

Problem: no console (headless), no peripherals, and it will only boot from the hard disk. It has four serial ports available and network connectivity.

Solution: load a modified Linux installer onto the hard disk. It must redirect the console to a serial port and be able to do a network install of the Linux distribution.

Here’s a summary of the steps I took to install CentOS on the appliances. If you’re interested in details, drop me a line.

  1. Download diskboot.img from the CentOS 5 distribution. This is a boot image most often used to boot from a USB key. We’re going to write it to the hard disk instead.
  2. Mount the diskboot.img file on a Linux system and edit the syslinux.cfg file. At the top, add:

    serial 0 9600

    which tells the installer we’re using serial port 0 (COM1) at 9600 baud. Change the “default” line to:

    default text

    because the terminal is initially screwed up so we won’t be able to make a choice; we’ll have to let it default to the right one.

    Now on “label text” add the following to the end of the “append” line:

    console=ttyS0,9600

    which will tell it to boot the kernel and use serial 0 at 9600 baud for the console.

  3. Save the file and unmount the image.
  4. Connect the hard drive from the appliance to the Linux machine, and determine its device name. I’ll call it “appliancedisk”.

    dd if=diskboot.img of=/dev/appliancedisk

  5. Now put the appliance’s disk back in place. Connect a terminal to serial line 0 (COM1) and connect the appliance to the network. Boot up and you can control the install process through the terminal, specifying a network location for the install media. Remember to keep the console=ttyS0,9600 line as a boot option so that the serial console continues to work after installation.

VoIP network monitoring with SmokePing

Previously I mentioned the possibility of using SmokePing to show loss, latency, and jitter on the PSU VoIP network. The current version has a “slave mode” which would allow dumb pollers to be deployed around the network for different perspectives on network performance. Additionally, SmokePing can be customized to test with any packet size (for example, 160 bytes for G.711 in 20 ms segments) to see the statistics for the type of data packets we’re most concerned about. We could even write our own probe modules to do real RTP rather than RTP-sized ICMP packets. More to come on this after some testing.

Try this Google search to get at some public SmokePing sites (maybe) monitoring VoIP: “smokeping targets” voip.

VoIP service monitoring: existing tools

To what degree can we monitor VoIP services with our existing tools?

I’ll be spending some time in the next few weeks/months evaluating VoIP service monitoring products exhibited at CIPTUG. These products are end-to-end solutions and can really give us a solid look at what’s going on in the VoIP network. But meanwhile, what (more) can we do with what we’ve got? The rest of this post is a lot of “thinking aloud.”

We’re using three main products right now to do the bulk of our system monitoring and trend-reporting: Castlerock SNMPc, Cisco Internet Telephony Monitor, and Cricket. I’m speaking mostly about monitoring and trend-reporting related to the VoIP system and network infrastructure that supports it. There are many other systems out there monitoring other parts of the network.

Castlerock and Cricket perform service monitoring by SNMP and configurable service polling. With these we can get any statistics exposed by SNMP as well as service availability information by connecting to a service’s network socket. This takes care of server vitals (CPU, memory, I/O, process stats), application availability (CallManager, SQL database, web services), and some network stats (up/down, byte counts). We’ve also got gateway channel usage.

Those are the low-hanging fruit and many of those stats are already being monitored and recorded. Next comes PBX statistics that will depend on how we can query CallManager and Unity using SNMP and Perfmon. I know these data are exposed but the integration into existing tools may be challenging. Other systems that may be added to the VoIP environment, such as Asterisk and OpenSER, will have to expose data in similar ways.

We can get network performance data including latency and jitter by using the Smokeping tool, which we already use to measure the core backbone.

What do we need to monitor at the endpoints? ITM tells us when endpoints disappear in groups (e.g. a network segment goes down). I’d like to be able to place automated test calls between two randomly-chosen endpoints for spot checking.

With current tools, we probably can’t get live call-quality information including mean-opinion score (MOS).

The third-party VoIP monitoring tools excel by putting all these data into one console as well as giving the live call-quality information. A piecemeal solution may turn out to be too cumbersome to use / put up with on a continuous basis.

CIPTUG 2007 exhibition floor and other notes

From the show floor:

  • VoIP system and network monitoring tools and suites vendors: Prognosis, NetIQ, NetQoS. NetQoS seemed to have the most thorough and least-intrusive (no probes in the network or services installed on CallManagers) monitoring suite. The size of Penn State’s network necessitates the use of a better toolset than just what is provided with Call Manager and the other applications. Tools from any of these providers as well as from Cisco (Operations Manager) could do the job; the biggest likely issues are usability and licensing costs.
  • Applications: Polycom demonstrated their application that allows the Cisco IP phone to control Polycom video endpoints. Very cool and an alternative to using Cisco’s video gear. Berbee and Syn-Apps had a lot to say and demonstrate about emergency response; they’re responding to post-VT requirements being laid out by organizations with large campuses. Citrix demoed their click-to-call application, which did not impress.
  • Swag/prizes: As for swag, APC gave out 256MB USB drives, Tripplite gave out some power testers and everyone gave out toys, goofy pens and t-shirts. Best prizes were from Zoom, who gave out three radio controlled helicopters, and GlobalKnowledge, who gave out three $500 Best Buy gift cards. INX is giving away an Xbox with Guitar Hero 3 tonight.

Some brief notes from seminars I attended:

  • From a seminar on management of Cisco Unified Communications, I really liked what I saw of the current version of Provisioning Manager. We evaluated a previous version of this but the current version (1.1) may be the time to review it again. Most important features are fast provisioning of phones, voicemail and services in one applications, delegation of privileges, and audit trail of changes. If you buy it as part of the suite of their other Unified Communications Management products, there’s some discount.
  • Cisco Unified Applications Environment is incredibly cool. It’s not just about IP phone services; it’s about applications that use all aspects of the UC environment. You can build a voice app, a web app, a hybrid, whatever. More information on Cisco’s site; SDK available on Sourceforge. (You can develop and test using the provided small number of licenses for free.)