I was hoping, when I first had the idea for this post, that it would be a run-through of all the steps I has taken in setting up my dedicated server. It was going to include links to the resources that had helped me along the way, and maybe a few hints and tips for other newbies doing the same thing.
It would appear, however, that whoever it was that hacked into my server and turned it into a spam-sending machine, had other ideas.
(This post is a bit long and rambling, so feel free to skip to the end if I start to bore you.)
A bit of background
This time last year, if you had asked me about running my own Web server, I would have told you that I had no interest in it. I was perfectly happy with the cPanel based shared hosting that I was been using, and didn’t see the need for anything else.
This all changed during my time at Propeller Communications, where I was introduced to version control. My first taste of a version control system was Mercurial, and while my own experience of it was rarely bad, and the bundled TortoiseHG was a joy to use, it didn’t take long for me to realise that the de-facto industry standard was Git (thanks, mainly to GitHub), so upon leaving Propeller, I made the switch.
The list of benefits afforded to users of version control is long, but the one benefit that really caught my attention was the ability to push changes I had made on my local machine directly to the server. No longer did I have to use FTP to upload the correct files to the correct place, while remembering to removing files that weren’t needed any more. I simply had to run one command, and everything was taken care of for me.
But, in order to reap the benefits of Git, I needed my sites to be hosted somewhere that supported Git – and to date I’ve yet to find a shared host that does. So it was time to say goodbye to cPanel, and say hello to SSH.
First tentative steps
Prior to setting up my live server, the one the hackers took a fancy to, I had built two local development servers.
The first, which was as much an experiment as anything else, was in built out of an old PC in the office at Ghost Design. The process involved booting off the Ubuntu Server (10.11) CD, selecting all the options I wanted (LAMP, DNS, SSH, etc..), then, after watching the progress bar complete, I installed Webmin to help manage it. This, I felt, went quite well, and allowed me to gain more confidence using the Linux command line.
The second, which I use as a development server in my flat, was built out of a PC that I had been using as a Windows-based media server. The install of Debian 6 was a much more involved process because I had decided to install all the software I wanted manually. I also avoided installing Webmin, as I wanted to learn how do things properly.
The success of these two servers had filled me with confidence, and so on the 8th December 2011, with a small loan from my mum, I ordered a dedicated server from Hetzner. Because a development server should be as close to that of the production server as possible, I went for Debian 6 again, and for the most part, followed the same instructions as I had for my home server.
Skip to the end
On the 30th January I received an email from Hetzner stating that an ISP had reported my server for sending spam. I forwarded this email to Phil, who suggested that an incorrectly configured mail server might be at fault. After removing EXIM, I thought that would be the end of it, but three days later I received another abuse report.
A full week later I was still receiving abuse reports, and crying out to Phil for help. I have no idea how he managed it (via the use of the occult no doubt), but he tracked down the culprit: a whole bunch of unexpected files located in three of the sites/vhosts I was hosting.
Two of the sites were based on WordPress. I vaguely understand how the open source nature of WordPress, combined with an out of date install and some lax permissions, could allow someone to search the source code for exploits, then search the Web for an exploitable server. But the third was a static HTML site, meaning whoever had done this had been able to get access to it from one of the other two sites, meaning, potentially, the entire server was compromised.
To stop the immediate issue of spam being sent, I had to turn the server off, and following Sheepy‘s advice, I’m going to “Nuke it from a great distance and start again”.
Prevention is better than cure
So what can I do differently to prevent this from happening again? I think my main issue was that of permissions. You can afford a level of flexibility, and a more relaxed attitude to permissions on a development server, because, for the most part, it isn’t accessible to the outside world. For obvious reasons, the same isn’t true of a production server.
I’m also going to make sure that any software I use on the server is kept up to date, thereby increasing the chance of exploits being fixed.
Anyway, I’m going to reinstall the server soon, and I’m still hoping to write the server newbie post, so watch this space.
