Follow us on...
Follow us on Twitter Follow us on Facebook
Register
Page 1 of 2 12 LastLast
Results 1 to 8 of 11
  1. #1
    ScottW's Avatar
    ScottW is offline Founder
    Join Date
    Sep 2000
    Location
    Leawood, KS
    Posts
    3,279
    Thanks
    0
    Thanked 17 Times in 11 Posts
    Blog Entries
    1

    Live FileVault and Sparse Bundle Backups in Leopard

    Mac OS 10.5 (Leopard) introduces the use of Sparse Bundles for FileVault. These are mountable directories full of banded data in 8M chunks. When you add, modify and remove files in FileVault, one or more of the band files will change, depending on where your data is stored.

    This opens up so many possibilities for making backups in a secure and quick fashion.

    For example, it is possible to backup your FileVault directory while you are logged into it. No, I'm not talking about the actual data and storing it insecurely somewhere else, or even on another sparse image or sparse bundle file, I'm talking about copying the FileVault sparse bundle itself. Using a handy included command line utility Rsync, I copied my 16GB mounted FileVault directory to another drive which took in the neighborhood of 15 minutes or so. I double clicked on it to mount it, and it asked for my password and to my amusement, it mounted without logging out of FileVault or doing anything other than just running the backup.

    If that wasn't handy enough already, I grabbed about 20 files I had on my desktop (so much for using Stacks) and tossed them into a new 'untitled folder.' I unmounted the copied FileVault directory, ran Rsync against comparing my live FileVault sparse bundle with the copied FileVault sparse bundle. This time, instead of 15 minutes to perform the backup, it took around 30 seconds, updating only the changed bands in typical Rsync fashion. I then crossed my fingers, swiveled around twice in my chair, and re-mounted the copied sparse bundle. It worked. My changes I just made to my live FileVault appeared on the duplicate.

    Feeling young and spry I decided to continue my fantasy adventure. While the copied FileVault volume was mounted, I made changes to my live FileVault desktop folder and then did another Rsync. About 30 seconds go by and nothing happened. I had hoped my changes would show up on the mounted file, but they did not. I really didn't expect it to work, but then again, I didn't expect copying a live (not logged out) version of FileVault would work. Note though, that I unmounted the copied FileVault sparse bundle and remounted it, and when I went to view the Desktop folder where I made my changes, the changes for a brief second showed up, then went back to the previous state. So, while there was some weird "corruption" going on, the sparse bundle still mounted. I then, unmounted the sparse bundle, ran another 30 second Rsync between my live FileVault and my copied FileVault, then remounted the copied FileVault and everything was back in working order and my change showed up exactly as they where on my live FileVault.


    What Can I Do With This You Ask?

    You can setup an automated task to run Rsync every 15 or 60 minutes, or every 24 hours to have a working mirror of your FileVault directory stored on another hard drive. Your backups are 100% secure, because you're not backing up the data files inside FileVault, you're backing up the encrypted files that make up your FileVault.

    Rsync was made to keep files in sync over a network connection. Whether you are running Apple File Sharing, Samba, or even an SSH tunnel, you can backup your FileVault directory securely to remote system, be it on your local network in your home or office, or over the Internet. For those looking to backup over the Internet, getting the first backup will be the painful process (when dealing with slower upload speeds on Broadband and DSL services) but from that point on, backups should be fast and painless, with only the amount of data needed to be transferred based on what changed.

    But, don't let your imagination stop there. Let's forget about FileVault and let's just say that we'd like to backup our iPhoto Library to an offsite location that we don't own and we want our iPhoto Library to remain encrypted from the eyes of others. All we would need to do is to create a Sparse Bundle with Disk Utility, use Rsync (or your favorite File Sync Utility) to sync your iPhoto Library with your local sparse bundled file. Then use Rsync to mirror the sparse bundle to a remote system. While this is a two step process, it does indeed work.


    Words of Caution

    Always test your backups. Never assume they will always work and be available to you. For your own sanity, log out and make a copy of your FileVault sparse bundle the old fashion way, prior to giving this a try. It will make me feel better about providing you the commands to work the magic discussed earlier and hopefully give you a little more freedom to play around. If you already have a solid backup of your FileVault directory, good for you, you may begin.


    How To Guide

    1) Open Terminal

    2) If your User directory is called "scott", and your backup drive is called "SnackFood", then you will want to make use the following command to make the first copy of your FileVault sparse bundle.

    Code:
    rsync -avE /Users/.scott/scott.sparsebundle /Volumes/SnackFood/
    Sit back and watch the files copy. The "v" option above in the "-avE" tells rsync to provide a verbose output, and let you know what the heck it is doing. The "E" option, we will only use on the first copy, this copies over special files that will make your copied directory show up as a double-clickable disk image, vs just a worthless folder.

    3) Once that is complete, go and mount the copied directory and if all went well, you should see an exact duplicate of your FileVault directory. Word of caution, SpotLight is hungry for data and once you mount it, it will start to chewing on it for fuel. It is best to add it to your Spotlight Private list as soon as you mount it, else you will be cussing at Spotlight when you attempt to eject your copied disk.

    If Spotlight beats you, you can do this to eject it if dragging it to the Trash can doesn't work.

    Code:
    hdiutil detach /Volumes/scott -force
    4) Make some changes to your FileVault file system, duplicate a few files or create new folders and move a few things around, then Rsync again, only using this command.

    Code:
    rsync -av /Users/.scott/scott.sparsebundle /Volumes/SnackFood/ --delete
    Caution: Syntax is VERY important with rsync, adding or not adding a tailing slash can mean something you did not intend to do. So, please follow my example. In addition the --delete will remote any files in the sparse bundle that have been removed and makes sure your copy is an exact sync of your original sparse bundle. In my tests, I didn't use the --delete function, and it would be safe to try it without it your first time to make sure your syntax is correct. However, you will want to use it for regular backups should you choose to ue this method.

    Dealing with spaces in file paths on the command line. If your hard drive is named "Snack Food", then in the above examples, you would have to put a backward slash in front of the space, this is called "escaping" the text.

    Code:
    rsync -avE /Users/.scott/scott.sparsebundle /Volumes/Snack\ Food/

  2. #2
    plsuh is offline Registered User
    Join Date
    Nov 2007
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts

    THIS IS A REALLY, REALLY BAD IDEA, DON'T DO IT. IT WILL ALMOST CERTAINLY LEAD TO A CORRUPTED BACKUP OF YOUR FILEVAULT SPARSEBUNDLE.


    When a disk image is mounted read/write, Mac OS X can and will rearrange files on it whenever it needs to. This often happens when an application requests space to write to a file. Such rearrangements will lead to writes to both the file space itself and the disk catalog. Since stuff is always being written to ~/Library/Caches and ~/Library/Preferences, changes are always occurring to the sparsebundle when it is mounted as a FileVault home directory.


    The author only tried mounting the image. He did not run hdiutil verify username.sparesebundle on it to check that the backup of the sparsebundle was truly intact. Even if it was intact once, there is no guarantee and in fact a strong likelihood that subsequent backups will be corrupted. There is a reason why Apple does not do portable home directory syncs of the sparsebundle while the FileVault sparsebundle home directory is in use. Setting up a sync to the file server using Sync Services would have been trivial, but the disk image corruption and sparsebundle band locking issues would have been much too difficult.






    --Paul

  3. #3
    ScottW's Avatar
    ScottW is offline Founder
    Join Date
    Sep 2000
    Location
    Leawood, KS
    Posts
    3,279
    Thanks
    0
    Thanked 17 Times in 11 Posts
    Blog Entries
    1
    I have been running this method, Rsyncing every 4 hours, every day since prior to publishing this article to present. To address the previous posters issues, hdiutil verify xyz.sparsebundle does not work, because it only verifies sparse disk images, not sparse bundles. Sparse bundles are a different monster. This method is proven to work and works flawlessly from my experience. Running diskutil verifyVolume on the sparse bundles show everything is in order.

    Just because Apple doesn't do this with Time Machine, doesn't mean that it's not a safe way to run a backup.

  4. #4
    Feuermurmel is offline Registered User
    Join Date
    Jan 2008
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    OMG, how dare you are, really suggesting such an abomination of a guerilla-backup method to innocent users and even worse, all this in good faith helping people. I have to reassure the previous concern and hopefully no-one who relies on their backup does follow this method.

    This is exactly why Apple tries so hard to hide the inner workings of all these technologies from the user, so they won't mess around this badly and still except it to work flawlessly.

    To address the actual problem; It does not seem as if you had a full understanding of how a file system works and how it is managed by the OS. Luckily this method will only break your backups and not the live data itself, I believe, but it will break. Believe me, I've seen enough file system getting damaged by using them in the intended manner and even so badly that Disk Utility couldn't repair it.

    Why do you think, the dialog that Mac OS X presents to you if you disconnect a external drive with a mounted file system is one of the rare occasions the red exclamation mark icon is used, accompanied by a very harsh and dismissive explanation of how badly the user screwed up. They're not joking by telling you that "recently saved data may be lost".

    If you want to get a feeling of what an effort it takes to organize all the files applications read and modify in your home directory run this command from the command line: `sudo fs_usage diskimages-helper'.
    Every of these activities change the internal structure of the file system inside the sparse bundle. To get a copy of your FileVault image that at least the journal should be able to repair reliably, the whole rsync operation must complete without any one of these operations interfering with the backup process.

    Maybe "it seems to work" is enough for you, but I have to remember this for the next recruitment…

    Feuermurmel

  5. #5
    masonk is offline Registered User
    Join Date
    Mar 2008
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks for this tip, Scott. I might be missing something, but it seems that rsync has no -E option. It's not in the man page, or on their website. And so I was not surprised when

    mason@big-spring~$rsync -avE /Users/.mason/mason.sparsebundle /Volumes/data/backups/
    rsync: -avE: unknown option
    rsync error: syntax or usage error (code 1) at main.c(875)

    What's up with that?

  6. #6
    grusaren is offline Registered User
    Join Date
    Nov 2007
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I have started to backup this way to see how it comes out, and it has worked perfectly for me. As a test I ran diskutil verifyVolume to see what happened at it returned an error: Incorrect number of thread records
    Don't know really what it means but it has not affected my ability to restore files.

    Question no 2: Why do you not use the -E flag on the second backup? Is it slower? I might be stupid here but I can't see the real difference.

    Otherwise, great post!

  7. #7
    ScottW's Avatar
    ScottW is offline Founder
    Join Date
    Sep 2000
    Location
    Leawood, KS
    Posts
    3,279
    Thanks
    0
    Thanked 17 Times in 11 Posts
    Blog Entries
    1
    masonk: Type "which rysnc" in terminal to see if you have more than one version installed.

    grusaren: The -E flag copies special files associated with the Mac. Once the this has been done the first time, you don't need to do it again because no new resource files will be created for that bundle.

  8. #8
    brianwebb01 is offline Registered User
    Join Date
    Apr 2008
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I've ran the initial rsync without errors however when I double click the sparsebundle disk image it verifies the image and says there was an error but I'm allowed to mount it anyway. Then it says there was an issue with the disk and I should replace it ASAP. After that I'm good to go. All files are there and accounted for. If I use diskutil verifyVolume it gives an error:

    mac ~: diskutil verifyVolume /Volumes/brian
    Started verify/repair volume (filesystem) on disk disk4s2 brian
    Checking Non-journaled HFS Plus volume
    Checking Extents Overflow file
    Checking Catalog file
    Checking multi-linked files
    Checking Catalog hierarchy
    Checking Extended Attributes file
    Invalid index key
    The volume brian needs to be repaired
    Error detected while verifying/repairing volume disk4s2 brian: Filesystem verify or repair failed (-9957)
    [ + 0%..10%..20%..30%..40%..50%..60%..70%..80%..90%..100% ]
    Finished verify/repair volume (filesystem) on disk disk4s2 brian
    Error detected while verifying/repairing volume disk4s2 brian: Filesystem verify or repair failed (-9957)

    Any ideas?

 

 
Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •