Oz, Linux, SD and Photo Rescue

Sometimes you realise that you really are an ubergeek.

My parents just spent a couple of months touring around Australia by plane, train and spaceship. En route they stopped off in Tokyo for a few days and were pleasantly surprised to discover that eating raw fish was actually quite nice. They were touting with them an Acer Aspire One Netbook running Ubuntu Netbook Remix upon which they were typing up their holiday diary and transferring pictures from the digital camera as they went along. Their holiday, and their first experience of using Linux were all going fantastically well right up until the last week of the trip…

I was in the pub enjoying not my first beer when I received an urgent gTalk message on my Android phone from Dad in Australia. He’d been transferring the latest batch of pictures to the wee Acer and something had gone Hideously Wrong. The camera couldn’t see them at all and the laptop could see some file names but they were all 0kb.

What were you doing? Just transferring them and then it stopped, I tried some stuff but I’m not really sure what to do… this desktop isn’t what I’m used to.

Two things; If the desktop had been Windows I’m not sure that would necessarily have improved matters regardless of how dedicated and experienced a silver surfer my Dad is. Second, the Netbook Remix of Ubuntu has a funky desktop that I’m not familiar with – or rather an interface thingy with all the buttons for programs designed to make things simpler. That’s fine until you want to ensure, through the medium of gTalk from the pub in Tokyo to your Dad in Oz, that he safely unmounts the SD before things get any worse. To this day I do not know how to unmount a disk properly in Ubuntu Netbook Remix when you can’t see the actual desktop nor get near a terminal – there should be a Big Friendly “Safely Remove Disk” button there somewhere and there isn’t, I’ll take this up with Canonical in due course.

Anyway, the disk had been unceremoniously inserted and ejected several times into the PC and the camera since the problem first arose so I figured the safest option was to have him take it out one more time and snail mail it to me.

The SD card arrived and sure enough there were some DSCFxyza.JPG files all of 0kb. There were also folders called “Documents” “Pictures” “Music” and “Videos”. This makes me suspect that the root of the problem was that the home directory on the Acer lives on it’s own SD card and that got pulled out to swap in the camera’s card and the OS got confused about what it was copying where. The cause though, by this point, is academic. Recovering the pictures and recovering my parent’s faith in Not Windows is everything!

There is only one picture which is actually intact and viewable and the rest are 0k though the disk properties suggest that the 1Gb SD is almost full at 892.5Mb in use.

Rather than risk causing more damage, I grabbed an image of the disk and set to work on that:

# sudo dd if=/dev/sda1 of=/home/username/diskimage

First off I grabbed and ran fatback on it, a tool for recovering various damaged FAT partition problems. This recovered much the same set of file names I could already see but sadly no more of their actual contents.

Next up is having a look at the one JPEG that is blessedly intact. A bit of digging reveals that it is an Exif format file. An hour of Googling later and I’ve discovered what the file format for Exif is. I also work out that an Exif image will start with the sequence of bytes (translating some into ASCII):

ff d8 ff e1 [xx] [yy] E x i f 00 00 I I * 00

Where xx and yy vary according to header length.

And will finish with

ff d9

As long as this is not followed by

ff db

Which would indicate the end of a section rather than the end of the picture (there’ll be multiple sections if the picture has an embedded thumbnail for example).

Now lets have a grep for the Exif opener ff d8 ff e1:

hexdump disk | grep -c "d8ff e1ff"

This at least suggests that there are a goodly number of headers intact somewhere in there and since this is FAT16 the files should be sequential… there is a chance I can grab the things back out of there!

So I hacked together a Python script to do the following:

  • Grab 16 bytes from the disk dump file into a buffer.
  • Check the buffer against the pattern that matches the start of an Exif file.
  • Get the next byte, drop the first one from the buffer and append this to the end.
  • Rince Rinse and repeat till you match an Exif header.
  • Start gathering the bytes you read into a separate picture buffer (non circular this time).
  • Get the next byte, drop the first one from the disk reading buffer and append this to the end.
  • Check the buffer against the pattern that matches the end of a file.
  • Rince and repeat till you match an end of picture.
  • Dump the bytes you’ve gathered in the picture buffer to disk and repeat from the top.

This rescued 818Mb of Jpegs in 336 files, just in time for Christmas.

I think the other 74Mb still on there are probably avi video files but without a template file to look at I can’t locate them yet. That said, once I get one it shouldn’t take much adjustment to the script to locate and recover those too.

For the really interested, here is the ExifJpegRescuer Python script, released to the world under GPL V3 or later.

3 Replies to “Oz, Linux, SD and Photo Rescue”

  1. Well done, my darling! I daresay all of the above makes sense to someone, but to me its just gobbledegook. However, you’ve rescued the pictures, and your mum and I are over the moon about that. Definitely worth a bottle of malt next time I see you! You are such a clever chap! (Shame you can’t spell rinse, but.)