Technical Insights > Network Visibility + Security

BOFHs, PFYs and Sys Admins

2020-07-30  |  8 min read 

For those about to SUDO, We Salute YouJuly 31, 2020 – a day that will live in infamy. The 21st annual Systems Administrator Appreciation Day is here. As a former IT guy, I had the pleasure of taking care of desktops and servers back in the day when it was actually pretty common to have desktops and fully appreciate what it is like to be deep in the trenches in IT.

I was actually in a pretty tough environment, a networking company startup that had some weird things going on. One of which was one half of the business, Engineering (note the capital E) ran on Unix, in this case Solaris, which was actually pretty good until the one day that NIS+ blew up, and then it was very bad indeed. The other half, overhead, dead weight and other corporate type stuff, ran on Windows.

Personally, since giving up on Novell (which was heartbreaking but necessary) I liked Windows (resistance is futile and all that), but there were strong feelings about choice of OS that verged on religion, both where I worked and elsewhere. Indeed, I recall one TGIF gathering where an otherwise mild mannered associate who worked for Sun almost came to blows with a fellow from Microsoft because Unix was better or something.

We had a period of rapid growth accompanied by a need to eat our own dogfood which resulted in some occasional downtime. It also resulted in interesting challenges like despite the fact we were out of switch ports on a particular switch (hardwired ethernet, yo) we had some engineers pitch an absolute fit when they were put on a different subnet than their coworkers, the thinking being that if the network went down for only one subnet, that it would be crippling to be unable to work if the rest of the team was up and running.

We had some interesting times with procurement. Partly due to the fact that the purchasing guy, who was the former business school instructor of the CEO, was corrupt and would buy equipment based on cheapness and/or kickbacks rather than suitability for purpose. Sure, IT had published standards – stuff like enterprise focused laptops with stores of spare parts and everything needed to do things correctly, which meant that this guy would buy something else, usually out of greed, sometimes out of spite, mostly spec’d wrong, hard to support or both.

One episode revolved around cheap clones of Sun workstations. While everyone knew they were not as good as the real thing, they were certainly cheaper and that if some of them died that was OK because you could swap out the dead ones and throw them away and still have plenty of money left over. Except nobody considered what to do when one of them literally caught on fire in the server room. That was fun. Suddenly it was OK to buy the real Sun stuff again.

One of the engineers who was supposed to be working on a key new RFC for MPLS (back when that was a new, shiny and not yet fully baked thing and not something to rip out and replace with SD WAN) became highly animated when issued a laptop that the corrupt purchasing guy had procured, probably because it had an 800x600 screen and he needed to write some long RFCs. Never could get anyone to use that laptop, but at least it came with a kickback.

Email was a constant problem. Again, half the company was on Outlook/Exchange and enjoyed the ability to book meetings, easily manage contacts and have a generally pleasant user experience. The other half, Engineering, took bizarre pride in running personal versions of sendmail on their desktops. This of course resulted in all sorts of ghastly problems with email due to misconfigurations, routing errors, user errors, desktops going down or being rebooted, but at least the macho men were able to use elm or pine or whatever was the cool command line mail client. Most of these problems were blamed on the window lickers in IT who couldn’t make email work.

Speaking of email, here’s where I slip in a subtle product pitch. We had a problem with Exchange users in one remote office. They could login, they could send mail, they could receive mail but they could not get attachments. Only impacted users in that one office. Same user in a different location? No problem. Hmmm, this is odd. After hooking up some taps and capturing some mail traffic, a feat greatly aided by having a visibility fabric in place powered by network packet brokers (more on NPBs here https://www.keysight.com/us/en/products/network-visibility/network-packet-brokers.html), the wizards in the network silo were able to figure out that somehow MTUs for the branch were set smaller than for the rest of the network, causing traffic such as Exchange packets with attachment data, to get dropped. Mystery solved.

Anyway, back to our networking startup. In an environment like that, you need some sort of comic relief. For me this was BOFH – the Bastard Operator from Hell. An evil, arrogant, fictional sys admin built of roughly equal portions of snark and passive aggression. A typical interaction would go something like this – user calls helpdesk, complains of a lack of disc space. BOFH takes the call, verifies the user is out of space. Clickety click, rm – rf. There, you have plenty of room now! Next!

Simon Travaglia is most frequently blamed for spawning the BOFH and his loyal sidekick, the Pimply Faced Youth (PFY). Simon’s website says he is based in Hautapu, in rural New Zealand where he is able to cater to a somewhat unusual fascination with old timey printing presses. He is also responsible for bringing a little joy into the lives of thousands of overworked and underappreciated tech workers who kept the ones and zeroes flowing.

Speaking of snark, one of my favorite IT publications, The Register, has an archive of BOFH stories here: https://www.theregister.com/data_centre/bofh/. Simon has a collection here as well: http://www.bofharchive.com/

For those of you about to SUDO, we salute you. And Simon, thanks for the well needed chuckles back in the day.