The Hacker Factor Blog

Earlier this week, I attended a presentation by Jochen Wolters titled "Back Up Your Data or Get Ready for Tears" (pdf). Usually the tech talks that I attend are extremely technical and aimed toward hard-core programmers and power-users. However, this talk was for generic users and not programmers. (I think I was one of three programmers there, including the speaker.) As such, Wolters gave an awesome presentation that cut out the technical details and told the audience what they really needed to know.

My own backup needs are an extreme case. I use RAIDs, auto-sync and auto-backup directories, and multiple media devices, as well as off-site storage. However, the typical computer user (mom and dad) don't need five-nines uptime (up 99.999% of the time).

Back to Basics

There are really just a few things that the typical person needs when it comes to computer backups:

• Backups. Kind of a "duh" thing, but if you don't have any backups then you will eventually lose data. Whether the loss is from a hard drive crash, power spike that fries your computer, or your cat deciding to lay on the delete key, data loss will eventually happen. The data you will loose may include your tax forms, family photos, your list of passwords for all of your online services, and more. For damaged drives, data recovery services are very expensive ($2000-$3000 per hard drive, and that assumes that they can recover the data). According to Wolters, some data recovery services even offer 24-hour suicide prevention hotlines. (People can get very emotional when all of their family photos are deleted.)
  
  
• Automated Backups. Your backups should be automated. If you are doing them by hand then you are probably not doing it regularly or often enough. Backups should be automated.
  
  
• Verified Backups. How do you know that your backups are working? If you have never tried to recover a file, then try it now! You don't want to be learning how to restore data after a crash. Instead, you want to be familiar with the backup software and know how to recover the system before you need to use the skill. (Think of it like defensive driving. You don't want to learn how to drive on ice during your first ice storm.)
  
  
• Offsite Backups. Be sure to have at least one backup kept off site. And "off site" does not mean in your attic, basement, or other room in your house. If your house burns down, you want your data safely stored outside the building. Consider storing a backup hard drive in a bank vault -- it will cost much less than a data recovery service.
  
  A good off-site location is any place where you already have access. This may be your parents house, the office, or a friend's apartment. (My pilot friend uses his airplane hangar.) But keep in mind: if you don't have free access to your friend's house, then you may have to wait before retrieving your backup. (E.g., if he is on vacation for 2 weeks and you are locked out, that means 2 weeks without access to the backup.)

Types of Backups

At bare minimum, you really need one type of backup: a full disk copy. This is a full copy of the bootable system. This way, if/when your hard drive or computer dies, you can just slip the backup drive into a working computer and be up and running. And since it is a full copy, you know that every application you need will be fully functional.

The second type of backup is an incremental. It just stores the files that have changed since the last backup. Depending on your needs, you may want a history of incremental backups. This way, you can recover a file as it existed a few weeks ago. If you only keep the most recent copy, then you may lose intermediate changes.

For my own extreme needs, I use external hard drives and only perform full backups. However, I have a bunch of these drives and I cycle through them. I can go back a month without a problem. For critical short-term data, I use system redundancy -- copying files between computers and storing iterative backups as needed. (For source code, I use Subversion for tracking changes. The full backup includes the entire subversion repository and history.)

Of course, some of the work that I do cannot be stored on backups. For example, third-party forensic data usually includes the stipulation to not keep additional copies. In this case, the data that I have is not the original data (I work from copies) and I have specific systems that are not backed up. But that's an extreme case and not typical for regular users.

For critical data that seldom changes (e.g., tax records for the previous years), I burn them to DVD. I usually burn two copies, just in case one gets scratched. Although DVDs usually have a shelf life of 7-10 years, that's perfect for taxes. (Although it varies by accountant, you are usually advised to not keep tax records longer than 10 years anyway.)

Frequency of Backups

Most people view backups as a bother. They won't take it seriously until after they lose a lot of data. My own extreme backup solution only came about after some bad experiences. For example, I had a critical hard drive die 10 years ago. Since then, I keep frequent full backups. Later, I lost some source code between backups. Now I use a source code control system and multiple-system synchronization with backup checkpoints.

There are really two factors to consider for your backup solution: (1) how much data can you afford to lose, and (2) how long do you want to be offline? For my own needs, I can lose up to 4 hours of work and it will take me up to an hour to recover. In the worst case, it will take me 24 hours to repair or replace whatever broke, but I will still be up and running while I wait for the repairs to complete.

Typical user needs are nowhere near as extreme. Your backup starts automatically around 1:00am and a full backup can be done by morning. Recovering from the backup may take just as long -- unless you use a bootable backup. If you do one backup a week, then you can lose as much as a week's worth of data. It'll hurt, but it won't be too bad.

Setting up the backup usually just requires two external USB hard drives (one for home and one for the off-site location -- and you swap them periodically), and getting some backup software. Many USB drives come with backup software, but I usually don't recommend using it. For Apple users, Time Machine is awesome software. For Windows users, your operating system comes with a backup system and a scheduler for automating it. (Right click on the drive icon. Backups are somewhere in the little menu.) For Linux/Unix users, 'rsync -auvf' is your friend.

Typical Needs

The audience at this presentation consisted of regular people. One guy was a carpenter. One woman was a writer. Another was a professional lecturer. Mostly small-office/home-office companies, and most did not have backup solutions. The few people who thought they had backups running were not sure what software they used, whether it was full or incremental, or even where the backed up data resided. If you don't have backups, you're not alone. Now is the right time to setup a backup solution that fits your needs.

There's a couple of random thoughts rumbling around my head... Rather than writing a blog entry on each, I decided to just mention them here.

Oiling The Machinery

Everyone is complaining about the oil gusher in the Gulf of Mexico. And everyone seems to have their own solutions. Use hair, use hay, construct a man-made barrier island, send down sludge, and more. British Petroleum has a couple of solutions lined up -- if one fails, then they will try the next. One of their solutions won't be ready until August! Some people think the government should take over the capping processes, but our government can't even pave roads without months of debate.

A few people are blaming Obama for this problem. (These are probably the same people who are upset that the Republicans lost the election and still watch Glenn Beck.) Frankly, we can't blame Obama for this one. Blame Bush? Sure -- he caused it by easing governmental regulatory oversight between 2006 and 2008. Obama only inherited this mess. And given other messes like Health Reform, Financial Reform, Immigration Reform, and Lobbying Reform... Regulatory Oversight Reform is just another item in the to-do list.

Anyway, I think I know the solution to quickly stopping the oil gusher. Congress should pass a resolution preventing BP from collecting any revenue until the gusher is capped and the cleanup is completed. Until both of those happen, any revenue received by BP should either go toward capping and cleanup, or be forfeited to the government and impacted states. If we cut off their revenue, then they will have an incentive for achieving a faster solution.

Google and SSL

Google recently released a beta of an SSL solution for their search engine. (https://www.google.com) They claim that this will improve privacy:

This secured channel helps protect your search terms and your search results pages from being intercepted by a third party. This provides you with a more secure and private search experience.

There's a few problems here. First, SSL is a placebo. From a security perspective, it does not add very much security or privacy. To gain security and privacy, you really need SSL with client-side certificates -- but Google isn't offering that.

Second, I find it ironic that Google is offering a security and privacy solution. I mean, they store every search, associate searches with user accounts, and cache personal information. So for them to be concerned about search privacy is just... funny.

Summer's Here

Summer vacation has clearly started. The number of malware and attackers scanning my web site for vulnerabilities has increased 10x compared to last month. Looks like the k1dd13z are out of school.

The uptick includes a significant increase in scans for WordPress vulnerabilities. Sample initial scans look like this:

2010-05-03 11:10:10 | 72.46.136.130 | GET /wp-login.php

2010-05-09 17:43:28 | 188.40.73.239 | POST /wp-admin

2010-05-24 17:52:37 | 213.231.27.46 | GET /blog/wp-includes/js/tinymce/wp-mce-help.php

2010-05-24 17:52:40 | 213.231.27.46 | GET /blog/wp-admin/upgrade.php

2010-05-24 23:24:35 | 69.245.165.224 | GET /blog/wp-login.php

Of these scans, it is the tinymce one that bothers me the most. This is a WYSIWYG editor and it has a history of remote access vulnerabilities. If you don't need it, consider removing it or locking down your htaccess file and web pages.

Arizona State Lottery

Arizona recently passed Senate Bill 1070. The law basically says that people suspected of being illegal aliens will be asked to provide proof that they are permitted to be in the USA. Failure to provide proof can lead to incarceration and/or deportation.

I'm not going take a side on whether this law is racial profiling or justified. (Let's leave that debate to the pundits and citizens of Arizona.) Rather, I'm looking at this from the hacker point of view. The first US Citizen that is arrested and/or deported under this law will have a heck of a lawsuit. Most likely, the victim will receive an out-of-court settlement as an apology because the case won't have legs to stand on if a provable citizen goes to court. Anyway, this law should be called the Arizona State Lottery because you too can become a millionaire overnight!

Many years ago, I used to go to auctions to pick up computers, peripherals, and other odds and ends. Although I would get the equipment for my own use, I'd still browse through anything that came with my auction winnings.

Burnt in monitor? What did it say?
Old ribbon in a typewriter? What was last typed?
Receipts in a cash register? I can't believe someone paid that much!

But the biggest gold mine was always the hard drives. 99% of the time, there wasn't anything incriminating. Old emails, random data files, and standard (usually unpatched) operating systems. Occasionally I'd find something disturbing... love letters between a faculty member and his student, porn collected by some grad student, or passwords stored in plain text for an entire department. (Most of the auctions I attended were for old university equipment. One time they even auctioned off a Cray!)

Stored Privacy

I always found these drives to be a fun way to try new forensic tools. Deleted files, uncommon (or old) partition formats, and corrupted data made the challenge fun. But it also made me keenly aware: never give out hard drives. If a drive fails, destroy it. If a drive exists in an outdated computer, destroy it. And if a drive is just too old to use, then destroy it.

Seriously: deleting the content is rarely enough. Someone who gets it could recover data. Formatting the drive may also not be enough. (Spinrite is an awesome recovery tool.)

There is always someone who argues about using multiple formats, like "What if I format it 27 times?" or "What if I overwrite the drive with a ton of random patterns?" While those might deter recovery, you can never be 100% certain. And even if the data cannot be recovered today, there is a chance that someone will create a new tool that can recover the data tomorrow.

Frankly, the best solution is a degausser and a drill press. There are even industrial metal shredders, but those are too expensive for most small-to-midsized businesses. Besides, having a drive destroying party is actually fun for the whole family! Here's a great contest: speed destruction. Who can tear apart a drive the fastest, without damaging anything? (My fastest is 4 minutes, 21 seconds on a Barracuda. I could have cut off a minute, but I damaged one of the heads.)

Necessary Equipment

The main things you will need:

• Set of various hex screwdrivers. This includes really small, metric hex screwdrivers.
  
• A small flat and Phillips head screwdriver.
  
• One really big, flat screwdriver (for prying when all else fails).
  
• A dremmel tool (for drilling out any screws if you don't have the correct tool).
  
• A bulk tape eraser, or set of strong Earth magnets (for erasing platters).
  
• A small tin for holding all of the loose parts.

Older drives (MFM, RLL, and even old ATA) usually use flat or Phillips screws. Most newer drives use 6-sided hex screws.

Tearing the drive apart really just means removing every screw you can find. And there are always 1-2 screws under the sticker labels. I usually run my finger across the label until I find the dimple where the screws are located. Then I cut through the label (just stick a sharp screwdriver into the dimple) and remove the hidden screws.

The disk platters usually have very tiny hex screws. Remove them and the entire stack will come apart.

Finally, there are two really strong magnets that control the head's position. These are GREAT for hanging stuff on the refrigerator. These magnets are usually mounted in metal brackets -- and removing them is a serious pain. If you can remove them, then wipe them over the platters to erase the disks. Otherwise, use a bulk tape eraser. (Use the eraser AWAY from all of the other small parts, because EVERYTHING magnetic will fly toward the eraser!)

If you just want to destroy the data, then consider shattering or drilling holes in the platters.

Waste Not Want Not

The first few drives I tore apart were a learning experience, but I threw everything out. Today, I just gotta use them for something. Initially I used the parts to make techie clocks. However, after the 10th clock, it got boring.


Now I'm into more functional artwork. Like this multi-level earring holder. (She really liked it.) I stripped a broken camera tripod for the base.


I've still got a couple of dozen hard drives than need dismantling. (My last auction purchase was a RAID. It was a great $20 buy, but now it is just too slow, too small, and too power intensive. Time to make art!) So my question is, what do you do with your old hard drives? And if you make things, what do you make?

March was insane... I still can't believe that I wrote as much as I did. Besides trying to blog 1-2 times a week, I also wrote...

New Book

Completed my 3rd book: Ubuntu: Powerful Hacks and Customizations. Technically, this is the 2nd edition of my 2nd book. (Long story about why the 2nd edition has a new name. It's one of those things that is outside of the author's control.)

When I first wrote Hacking Ubuntu, the focus was on Ubuntu 6.06 (Dapper Drake). However, Dapper is past its support for the desktop edition and the server edition has about a year of support left. Also, with each Ubuntu release there are major changes. While the hacks worked well for Dapper, a few had problems with Hardy Heron and some didn't work for later releases. With this new book, I fixed the hacks so they would support Dapper Drake, Hardy Heron, and Karmic Koala. They should also be good for the next Ubuntu versions (but since Lucid Lynx and Maverick Meekat have not yet been released, I cannot guarantee everything).

One would think that revising an existing book would not be as time consuming as writing a new book from scratch. However, that really isn't the case. It takes almost the same effort to revise an entire book for the latest operating system releases.

Documentation

I've been finishing up the documentation for the photo and image analysis software. The software is geared toward law enforcement. I am not planning a general, public release because the learning curve is just too steep for the average person. Instead, the release will be initially limited to US law enforcement. (I'll probably change my focus to non-US law enforcement later this year.) When investigators testify on the witness stand, they cannot just say "I pressed a button and it drew a picture". They actually need to understand the algorithms. So my technical documentation explains the algorithms, provides sample use cases, and details the known limitations. I finished the writing last month -- over 100 pages for the technical documentation plus a 16-page guide for common scenarios and an 8-page installation guide.

I still have a little more work to do with the documentation. My friend, Chris Hanson, has been a godsend -- he's provided me with great example images. Unfortunately, I will need a few more pictures. Specifically, I need a few 100% computer generated images that are professional and high quality. If you are a professional graphic artist and can make high quality images similar to those found at the CG Society, please let me know! (I don't need custom graphics, exclusive rights, or a transfer of copyright. I'm hoping to find an artist who already has a sample portfolio and is willing to let me include one or two images.)

Papers and Code

I also wrote a bunch of white-papers including two that were over 100 pages. In total, I wrote nearly 200 pages last month and completed over 500 pages in papers, reports, and a book.

Not everything I wrote was in English. I did a ton of programming -- sloccount claims I wrote 3 months of code in one month, using C, Perl, Shell Script, and HTML. And while my college didn't recognize C or Perl as a foreign language, I think they really should count.

The last few weeks have been a serious rush. I think I can summarize it simply: newer isn't always better.

Spinning Down

A few months ago I lost "yet another" hard drive. Fortunately, it was part of a RAID, so I didn't lose any data. (A lesson I learned from my first hard drive failure -- always use a RAID.) I seem to be getting 2-3 years out of newer hard drives, and it does not matter which manufacturer created the drive.

I have a few old computers collecting dust in the back room. Recently I had a need for some software that I wrote back in the 1990's. I couldn't find a copy on my newer systems, but I knew it was on the old, dusty box. I plugged it in, powered it on... and it came up without a problem. Now, to put things into perspective: the hard drive is a 120 MB (yes, megabyte) Conner drive. I acquired it around 1990. This drive ran continuous duty for over 15 years before being powered down and archived for five years. And... it powered back up without a problem.

When it comes to hard drives, I plan for new ones to fail -- because they will fail. But old hard drives? I think my Conner could easily do another ten years continuous duty. (Too bad it is only 120 MB!)

Broken Windows

The newer X-Windows server (since about 2008) is much more automated. In Ubuntu's Karmic Koala (9.10), it does not even include an Xorg.conf file -- the entire configuration is automatically detected.

The good news is, the X-Server will likely configure itself correctly and start up without a problem. The bad news is, if it has problems, then many of the debugging tools that you will need are broken. Making matters worse, they have been broken for years.

A good example is the xvidtune program. If you have a flat screen monitor, or even a newer tube monitor, then it will likely auto adjust the frequency and center the image on the screen. But if you have an older monitor, then you may need to manually align the desktop's position on the display. Depending on the video card, monitor, and auto-detected X-Windows settings, the desktop may need more shifting than the monitor's manual controls allow. The real solutions is xvidtune, which allows you to adjust the position on the display by tweaking the horizontal and vertical frequencies.

Unfortunately, xvidtune has been broken for years -- since X-Server version 1.4 (2007). And while plenty of people have reported the problem, it has remained broken for at least three years.

HTML Doc

I've been doing a lot of technical documentation lately. I'm writing it in HTML and using htmldoc to convert it to PDF. The problem is, my older Ubuntu Dapper Drake system could generate the docs but all of my newer systems could not. It turns out, my HTML includes arrows for menus (–› created using –›). On the newer systems, they just print blank spaces.

I eventually traced the problem to the version of htmldoc. Version 1.8.24 works fine, but the newer versions (1.8.27 through 1.9) seem to have problems with ampersand codes.

Et Tu, JPEG?

For my image analysis stuff, I rely on the FreeImage library for loading most image formats and saving all formats. (FreeImage has a few quirks with corrupted files, so I wrote my own libraries for loading some file formats.)

I recently upgraded from FreeImage 3.11.0 to 3.13.1... and immediately noticed some problems. The Error Level Analysis and color space algorithms were giving different results for some of my regression tests. I even tried 3.12.0 and 3.13.0 -- and found the cutoff: 3.12.0 renders JPEGs correctly, 3.13.0 does not.

FreeImage actually uses the library provided by the Independent JPEG Group (IJG). FreeImage 3.12.0 uses jpeglib v.6b, while 3.13.0 upgraded to jpeglib v.8. Somewhere between 6b and 8, IJG did a significant rewrite to their library for applying chrominance. The net result: JPEGs rendered by IJG's jpeglib v.8 no longer look like JPEGs rendered with other libraries (IJG and non-IJG).

Don't get me wrong: The pictures still look like pictures, the differences are subtle, and the changes really only impact extreme corner-cases. However, if the library does not render colors in those corner cases exactly like other libraries, then I cannot use it. Good thing I could easily regress to 3.12.0.

Blast From The Past

Not everything old is better than their newer counterparts. My iPod is a much better MP3 player than my old no-name brand player. My USB LED mouse is far superior to the old serial mouse (if for no other reason than the wheels don't get gummed up). And my netbook is a huge improvement over my old Dell laptop.

But in the last few weeks I have been repeatedly reminded that newer is not always better. (And don't get me started on the Toyota recall. Good thing my car is old...)