Over the last few months I have had friends and associates contact me about hacked web sites. In each case, someone (or something) planted hostile URLs on their web pages. These URLs would redirect visitors to porn sites or serve up viruses. Worse: these URLs would be embedded everywhere -- in HTML, in PHP, and in back-end databases.
The question they always ask me: What should I do?
It is easy to tell people that they should have a disaster recovery plan in place. However, few people have one. Other
pre-attack advice, like hardening servers, changing defaults, and installing filters is great advice, but is usually ignored. In my experience, the sites that have taken simple steps and have plans in place are not the ones usually compromised. The common compromises are directed at non-technical users who installed default software and ignored even basic maintenance.
Post-Compromise
So let's say you have a default WordPress or Wiki or Blogger installation. It isn't a question on whether your site will be compromised or infected. The only question is
when. And like most people, you haven't maintained your software (applying patches, upgrading as needed), don't have backups (your ISP does that, uh, right?), and haven't removed default files or hardened the system. What should you do after a compromise?
There are plenty of good checklists out there. Some examples include:
While each of these sites gives good advice, there is no single consensus regarding appropriate steps. My own checklist is a little more detailed and extreme.
Neal's Post-Compromise Checklist
Nobody wants to have their site compromised. However, like auto accidents, bad things happen. If you were not paying attention (like texting while driving or not applying system patches) then bad things are more likely to happen to you.
Here are the steps that I usually recommend to people with compromised web sites:
- Stay calm. Only the WordPress checklist included this advice, but it is very valuable. Now is not the time to panic, place blame, or get angry. Compromises and exploits go hand in hand with technology. Don't panic, deal with it.
- Check your own systems for malware. There is no point in fixing the server if your own workstation is infected. It is actually very common for a home computer to be infected and used to gain access to your public blog or web server.
- Take the site offline. Most e-commerce companies really don't like this idea. However, which is more important? Making a sale, or compromising your customer's credit card and tarnishing your reputation because your site was hijacked? Shut it down. Put up a temporary "We're upgrading, back in 48 hours" message.
After you take your site offline, check to make sure it really is offline. Some attacks actually hijack your domain name (DNS entry) and not the server itself. If your site still looks online, has the DNS server been compromised? Your DNS registrar and/or domain hosting provider can usually help you in this situation.
- Make a full backup. This includes all files, scripts, and database records. Yes, it is infected. But if you don't have backups then this is your only option.
- Grab your logs. You will need these to identify how the attack was done, when it happened, and who else might have been infected. This includes system logs, web logs, and any other kind of log file. Grab it first and see if you need it later.
- Evaluate the compromise. What kind of attack was it? Were they after "a site" or "your site"? Was the attacker looking for low hanging fruit or was it personal? Most malware, like the kind that inserts links, are automated. They scan for known vulnerabilities and infect anyone they find. If you were compromised this way, then it is likely because you have a default configuration with a known vulnerability (known to the attacker).
Defacements may be automated or semi-automated. They scan for sites with known vulnerabilities and then they either automatically or manually deface the site.
E-Commerce theft is usually associated with an initial automated vulnerability scan. The scan is followed by a manual compromise that is customized for the site. However, if you use a very common e-commerce package, then the compromise may be semi or fully automated.
Personal attacks are always manual compromises.
It is important to recognize that automated attacks are almost never against custom code. They look for known vulnerabilities in default installations. If you change the defaults, move default files, or otherwise filter and harden the site, then automated attacks are very unlikely to succeed.
- Look for similar attacks. Are other people running the same software getting attacked? Perhaps you need a patch. Is it everyone on the same server or in the same hosting environment? Maybe someone should maintain the system. Is it all of your accounts? Perhaps your computer has a virus.
- Change passwords. Between taking the server offline and changing passwords, attackers will be kept out while you repair the system. (Hopefully.)
- Wipe the system. There is only one thing people hate more than being told to turn off an e-commerce site, and that's being told to wipe the system and reinstall from scratch. But seriously, the inserted URLs and malware may only be the part that you notice -- much more may have been done to the system. There could be backdoor software, trojans, or embedded viruses that cannot be removed by a simple system restore. By wiping and reinstalling, you ensure that all malware is gone.
- Patch. Bring the system up to the current state of the art. While you're at it, harden the system and change all system defaults.
- Restore. You do have backups from before the compromise, right? If not, then install basic software (like blogs and wikis) and harden them first. Then place custom software on the system. Finally, restore content. Be sure to validate that the content is not infected. You can do this by reviewing the content before uploading it. (What? Review your 10,000 blog entries? Yes. There is no point in removing the malware from the server if you're just going to upload it again.)
- Change passwords again. Passwords before the patch and restore could have been compromised.
- Watch. Now you can turn the system back on. If the attackers come back, then you didn't patch or restore something. (And now you have experience to recover much faster!) Watch your logs and IDS and try to determine how they exploited your site. If the logs show nothing, then you know which parts of your site were not responsible for the attack.
- Blame. This is everyone's favorite part. If you don't have logs, don't regularly patch or update, and don't maintain the system, then you cannot blame anyone except yourself. Security is a moving target -- software that was secure yesterday may not be secure today. Unless an administrator did something completely stupid, such as posting login credentials in a public forum or actively assisting the attacker, then there probably is nobody else to blame. Blame the management.
Too many times I have seen management blame developers or software for compromises. For example, if your old version of WordPress was the source of the compromise, then they will blame WordPress even when newer versions are available. (Let's blame the software instead of ourselves for failing to maintain and harden the systems.)
Having your site compromised isn't fun, but it isn't the end of the world either. Stay calm and address the problem. Treat it as you would any other learning experience.
