{"id":24,"date":"2008-03-01T13:25:29","date_gmt":"2008-03-01T21:25:29","guid":{"rendered":"http:\/\/porkrind.org\/missives2\/?p=24"},"modified":"2013-05-30T01:48:50","modified_gmt":"2013-05-30T08:48:50","slug":"time-machine-and-a-linux-server","status":"publish","type":"post","link":"https:\/\/porkrind.org\/missives\/time-machine-and-a-linux-server\/","title":{"rendered":"Time Machine and a Linux server"},"content":{"rendered":"

I have a Debian GNU\/Linux server with a large raid 5 disk. I recently upgraded my RAID disks and suddenly have ample free space. I put OS X 10.5 on my MacBook back in November but didn\u00e2\u20ac\u2122t have enough space at the time to try out Time Machine. Getting Time Machine to work with my Linux server was annoyingly hard\u00e2\u20ac\u201cthe default Debian server doesn\u00e2\u20ac\u2122t support Leopard out of the box and Time Machine itself doesn\u00e2\u20ac\u2122t support non-apple file shares. Not to mention Time Machine seems slow and completely bug ridden (I\u00e2\u20ac\u2122m generally unimpressed, but when it finally works it\u00e2\u20ac\u2122s nice). Anyway, here\u00e2\u20ac\u2122s what I had to do to get it to work.<\/p>\n

Step 1: rebuild netatalk with ssl support<\/h2>\n

Apple, starting with Leopard, won\u00e2\u20ac\u2122t connect to appleshare servers that don\u00e2\u20ac\u2122t support SSL (encryption). This is a good thing, really. What is annoying is that Debian doesn\u00e2\u20ac\u2122t ship their netatalk package with encryption enabled (there\u00e2\u20ac\u2122s apparently some sort of licensing mismatch and they\u00e2\u20ac\u2122re fairly pedantic about those things\u00e2\u20ac\u201cwitness \u00e2\u20ac\u0153iceweasle\u00e2\u20ac\u009d).<\/p>\n

If you are familiar with Debian packages, the key thing is rebuilding the netatalk package like this:<\/p>\n

DEB_BUILD_OPTIONS=ssl debuild<\/pre>\n
If you are not familiar with Debian packages, first google for \u00e2\u20ac\u0153debian ssl netatalk\u00e2\u20ac\u009d or \u00e2\u20ac\u0153ubuntu ssl netatalk\u00e2\u20ac\u009d. There\u00e2\u20ac\u2122s a bunch of people out there with instructions. They basically boil down to:<\/p>\n
apt-get source netatalk\r\nsudo apt-get build-dep netatalk\r\nsudo apt-get install cracklib2-dev\r\ncd netatalk-2.0.3\r\nDEB_BUILD_OPTIONS=ssl debuild\r\nsudo dpkg -i ..\/netatalk-*.deb<\/pre>\n
At this point you should be able to mount an your Linux server on your Mac.<\/p>\n
Step 2: Getting The Mac To See The Server<\/h2>\n
Time Machine, for some stupid reason, doesn\u00e2\u20ac\u2122t seem to want to see network shares from non-Apple servers. There\u00e2\u20ac\u2122s a well known secret preference (how\u00e2\u20ac\u2122s that for an oxymoron) that you have to set from Terminal:<\/p>\n
defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1<\/pre>\n
You can make sure it\u00e2\u20ac\u2122s set with this command:<\/p>\n
defaults read com.apple.systempreferences TMShowUnsupportedNetworkVolumes<\/pre>\n
It should print \u00e2\u20ac\u01531\u00e2\u20ac\u00b3 if it is enabled. With that set, you should now mount the drive you wish to back up to with Finder, then go to Time Machine\u00e2\u20ac\u2122s Preferences. If all is well then your network drive will show up when you press the \u00e2\u20ac\u0153Change Disk\u00e2\u20ac\u00a6\u00e2\u20ac\u009d button.<\/p>\n
If the drive does not show up then you are going to have to mess around. I had a terrible time getting it to show up on a computer with 10.5.2 installed. One thing to try is adding a file to the root of the network share called \u00e2\u20ac\u0153.com.apple.timemachine.supported\u00e2\u20ac\u009d. Also try deleting the .0016cbcbc4c8 file in the root of the share (that number is for my computer\u00e2\u20ac\u201cyour computer will have a different number).<\/p>\n
Step 3: Try a small backup<\/h2>\n
Go into Time Machine preferences and hit the \u00e2\u20ac\u0153Options\u00e2\u20ac\u00a6\u00e2\u20ac\u009d button. Keep adding excluded directories until only a small part of your disk is set to back up. Right click on Time Machine in the Dock and select \u00e2\u20ac\u0153Back Up Now\u00e2\u20ac\u009d. If all goes well then Time Machine will chug along for a while \u00e2\u20ac\u0153preparing\u00e2\u20ac\u009d (going through your whole disk sizing everything up), then doing the actual backup, then \u00e2\u20ac\u0153finishing\u00e2\u20ac\u009d (Deleting interim backups\u00e2\u20ac\u201cif you have a lot of these then finishing can take a really really long time).<\/p>\n
If it gets an error then you might need to create the sparse bundle for it.<\/p>\n
An aside\u00e2\u20ac\u201csparse bundles and Time Machine<\/h3>\n
A \u00e2\u20ac\u0153sparse bundle\u00e2\u20ac\u009d is a new format of disk image added in OS X 10.5. This follows a long line of disk image formats from the old \u00e2\u20ac\u0153.img\u00e2\u20ac\u009d of System 7 to \u00e2\u20ac\u0153.dmg\u00e2\u20ac\u009d introduced in OS X.<\/p>\n
Sparse bundles are actually pretty neat. They are almost the same as \u00e2\u20ac\u0153sparse image\u00e2\u20ac\u009d \u00e2\u20ac\u0153.dmg\u00e2\u20ac\u009ds from previous versions of OS X. Traditional disk images allocate all the room up front. So if you create a 10GB disk image then it would create a 10GB file on your disk.<\/p>\n
The sparse image is different\u00e2\u20ac\u201cif you create a 10GB sparse disk image then the file that is created on the disk is actually much smaller, basically just the size of the HFS+ filesystem. When you add files to the disk image then the .dmg file itself grows to accommodate new data.<\/p>\n
Sparse bundles work the same way except that instead of one gargantuan 10GB file (assuming you fill up the whole disk image), the data is spread out in a large number of small \u00e2\u20ac\u0153bands\u00e2\u20ac\u009d. Each band is a file that is a specific size (8MB for Time Machine sparse bundles) that holds just one little section of the disk image\u00e2\u20ac\u2122s data.<\/p>\n
Why is this good? I\u00e2\u20ac\u2122m glad I asked! It\u00e2\u20ac\u2122s particularly nice when you are syncing the image between 2 computers. Say I mount a normal disk image and change one letter in a TextEdit document. If I try to sync the disk image to another computer I basically have to copy the whole thing over (barring some cool rsync-like behavior). With a sparse bundle, I probably have to only copy one or two bands of data across. According to what I\u00e2\u20ac\u2122ve read, Apple create the sparse bundle format so that File Vault was more compatible with Time Machine (it can back up bands instead of entire disk images).<\/p>\n
Anyway, when you back up to an external FireWire or USB disk, Time Machine backs up directly to the disk. When you back up to a network disk, Time Machine puts a sparse bundle disk image on the network share and then backs up to the disk image. This theoretically allows them to back up to a network filesystem that by itself doesn\u00e2\u20ac\u2122t have the necessary capabilities for Time Machine. NFS, or SMB, for instance should<\/em> be able to be able to host Time Machine backups with no issues\u00e2\u20ac\u201cI haven\u00e2\u20ac\u2122t tried this though.<\/p>\n
The problem is that OS 10.5.2 seems to have a bug where you can\u00e2\u20ac\u2122t always create a sparse bundle on a network share. If you try to do this with Disk Utility you will get an \u00e2\u20ac\u0153operation not supported\u00e2\u20ac\u009d error.<\/p>\n
The solution is to create a sparse bundle on your disk that is the size of your network backup disk and then copy it over. My backup disk is 2.25 TB, so I\u00e2\u20ac\u2122d create a 2.25 TB sparse bundle disk image. Once you copy it over to the network drive you can delete it from your local disk. I used this technique to get my dad up and running with his shiny new Ubuntu server.<\/p>\n
By the way, for this to work you have to name the sparse bundle disk image correctly, which takes a little trick. Start a Time Machine backup and look in your network disk. Time Machine will have started creating a sparse bundle on the disk. Copy the name of this sparse bundle before it gets the error and deletes it on you. Mine is called \u00e2\u20ac\u0153black_0016cbcbc4c8.sparsebundle\u00e2\u20ac\u009d (my computer name is \u00e2\u20ac\u0153black\u00e2\u20ac\u009d). Either create the sparse bundle with that name or rename it once you\u00e2\u20ac\u2122ve created it.<\/p>\n
Other issues<\/h3>\n
My computer behaved very, very strangely before I got everything running smoothly. One thing it did was stick in the \u00e2\u20ac\u0153preparing\u00e2\u20ac\u009d phase for huge periods of time (24 hours, sometimes). I\u00e2\u20ac\u2122d just stop the backup and let it try again later. After a couple tries it would mysteriously work.<\/p>\n
Another thing it did was get confused with the progress. It would start up a backup and say something reasonable in the progress bar (0GB completed out of 30GB). Everything moved along but when it hit the \u00e2\u20ac\u0153end\u00e2\u20ac\u009d it would suddenly start incrementing the total size: \u00e2\u20ac\u015332GB completed out of 32GB\u00e2\u20ac\u009d. And it would just keep growing and growing. It would reach ludicrous numbers sometimes, saying \u00e2\u20ac\u01532TB completed out of 2TB\u00e2\u20ac\u009d while backing up my 250GB disk. Sometimes it would complete on its own and sometimes I just stopped it after 24 hours.<\/p>\n
Interestingly, both of these strange symptoms might be related. I captured both of these situations using 10.5\u00e2\u20ac\u2122s dtruss and found that in both cases if was seemingly looping through my disk multiple times. I would see the same files and folders being iterated through over and over again, like there was something wrong with my filesystem. I did a fsck with Disk Utility, though, and it reported that everything was fine.<\/p>\n
So I think both of those issues are just stupid bugs in Time Machine (or one stupid bug hitting in two places). It\u00e2\u20ac\u2122s times like these when I curse Apple\u00e2\u20ac\u2122s crappy proprietary software. If this were a linux app I\u00e2\u20ac\u2122d just download the source and fix it myself. But the Bastards at Apple think it\u00e2\u20ac\u2122s better for Time Machine to be a %$#@! black box. Which is OK when it works, but really sucks when it doesn\u00e2\u20ac\u2122t work. <\/rant><\/p>\n
Step 4: Slowly remove excludes<\/h2>\n
To get around these issues I would alternately let it run for huge long periods of time and then stop it if it were running for more than 10 minutes. I also excluded almost all my disk in the Preferences (as I mentioned before) and then slowly removed exclusions until my whole disk was being backed up.<\/p>\n
I\u00e2\u20ac\u2122m at the point now where Time Machine just mounts up my network drive, backs up and only takes a couple minutes to finish. Getting there was slow (it took me about a month to get to this point!!!) and even though it is fairly stable now, I still can\u00e2\u20ac\u2122t help feeling like Time Machine is a buggy pile of crap.<\/p>\n
Some helpful links:<\/p>\n