One of the internal dev servers at work got bitten by this old Jenkins bug that I noticed a month or so because it popped up in the firewall logs. Because this isn’t a production machine and is mainly used as an internal testing server (it’s also running a pmta server which is pretty scary), we just shut it down and disconnected it from the network. I assumed the developers would probably torch the entire machine and start over. They didn’t and I had to find the offending lurkers and rescue this box. I’ll admit that I did it the laziest way: I just installed Sophos and waited for the client to either confirm that I had an admittedly tentative false positive or start crap bombing the cloud console with notifications. The crap bombing commenced about 75 seconds after the client piece was finished installing. It managed to open about 12 cases in our ticketing system before I could even disconnect the network.
Once the AV picked up on it I at least knew which file was being obnoxious and could go clobber it and make sure it hadn’t set up any cron jobs to reestablish itself afterwards. The path was something like /tmp/.jenkins/jenkins/-bash. Yup: -bash. It was the end of a pretty long day and it took forever for me to remember how delete a file like this in bash. I hadn’t actually considered that when I first started sawing on it. The remember this idiot from this particular situation is that bash assumed everything that begins with a hyphen is going to be an argument. I futzed around with quotation marks and backticks for a while and got nowhere but frustrated. The oddly easy answer was this: rm -v — -bash. That didn’t come from memory or anything like that but a frustrating round of trying to google up a solution with every query I could think of that didn’t actually use a hyphen which becomes an exclusion argument for that search. So, the short story is that everything after the double hyphen becomes a filename that bash no longer cares about evaluating. In retrospect, I probably should have renamed it and taken a look at what it was actually doing. It was the end of a long day though so I just nuked it from ssh. It was the only way to be sure. Actually, I wasn’t ever sure so I yanked the network again and kicked off an AV scan before I left. A learning experience, yet again, that wasn’t a whole lotta fun especially when my load average is above 4 on an already aging server that I apparently need to preserve.
