Time Machine sparsebundles have suddenly stopped working

[rbpeirce]

I am running Monterrey 12.6 on my MBP. For years I have done time machine backups of my wife's MBP to a HD attached to my computer. This has always worked well so I haven't paid much attention other than making a copy once a month to another HD just in case.

For some reason I happened to check and discovered the last backup was 9/27. I did a "backup now" and it couldn't open the sparsebundle. I tried opening it from my computer and it didn't open. I then tried the copies and only the oldest would open. Other sparsebudles open fine. All the copies also opened the last time I checked.

Other than updating from 12.5 to 12.6 last month, a week or more before 9/27, I can't think of anything else that might have changed.

Show package contents works but open with DiskImageMounter.app (the default) fails. Any ideas?

 

[rbpeirce]

I got it working but I'd still like an idea of what happened. Are sparsebundles just unreliable?

I restored from the copied backup that worked. My wife's machine recognized it and updated it. Now I'll have to check more often to be sure it is still working correctly

 

[rbpeirce]

I haven't said anything about this since last October but I'm still curious. I use Time Machine to backup my wife's MBP over the network to a sparsebundle on a HD attached to my MBP. In my experience things either work all the time or fail all the time. This fails randomly. I may get several backups one day and several days without any backups. I don't know why.

I copy the sparsebundel to another drive for redundancy and I check to be sure it opens. So why is Time Machine sometimes failing and sometimes not?

 

[MacInWin]

So why is Time Machine sometimes failing and sometimes not?

Sparebundles are very highly compressed files. Basically if one bit (not byte) goes wrong, the entire sparsebundle cannot be used and has to be discarded and started over. AFAIK, there is no repair process for them.

The solution? Stop using TM over the network. Get a drive and directly attach to her machine for backups.A rotating drive will do.

 

[rbpeirce]

Sparebundles are very highly compressed files. Basically if one bit (not byte) goes wrong, the entire sparsebundle cannot be used and has to be discarded and started over. AFAIK, there is no repair process for them.

The solution? Stop using TM over the network. Get a drive and directly attach to her machine for backups.A rotating drive will do.

I know about the bit. That's why I copy it to a backup drive each day and check that it opens. If it doesn't I restore from the last good copy.

Attaching an external drive to her computer is not a viable solution. I considered that but the way she uses her computer makes it impractical.

Network is the only practical way and it always works but not with 100% reliability. That's what is confusing. I may get five backups one day and no backups for five days. I'm trying to figure out what to do about that. Possibly nothing can be done and we just have to live with it.

 

[Slydude]

The solution? Stop using TM over the network. Get a drive and directly attach to her machine for backups.A rotating drive will do.

Does that still work? I thought the snapshot files TM now uses were actually a form of sparsebundle file.

@rbpeirce Just a couple of questions for clarification purposes. My apologies if you have already answered this and I missed it.

1. Are both computers running Monterey? I'm not sure that matters but I thought it was prudent to ask just in case.
2. Are you seeing any error messages when the backups fail?
3. Is there a pattern to the failures? For example, are the failures occurring with the original Time Machine backups or the copies that you create?

 

[rbpeirce]

Does that still work? I thought the snapshot files TM now uses were actually a form of sparsebundle file.

@rbpeirce Just a couple of questions for clarification purposes. My apologies if you have already answered this and I missed it.

1. Are both computers running Monterey? I'm not sure that matters but I thought it was prudent to ask just in case.
2. Are you seeing any error messages when the backups fail?
3. Is there a pattern to the failures? For example, are the failures occurring with the original Time Machine backups or the copies that you create?

I'm currently on Ventura. I think she is still on Monterey but this goes back to when were were both on Monterey.

No error messages. Often you can see a backup is in progress but it silently fails.

No pattern I can see. I have restored backups a couple of times but the randomness doesn't change.

 

[MacInWin]

Try getting T2M2 from here:

XProCheck, T2M2, LogUI, Ulbow and log utilities

XProCheck – check on XProtect Remediator scans completed and reported in the log Updated! Inspects the Unified log for a period of 1-30 days and reports all completed and reported anti-malwar…

eclecticlight.co

Use it to see what is going on in the logs on both machines.
SLY, there are differences between sparse bundles and snapshots. If you do a search at Eclecticlight.com, there are dozens of articles on each. The biggest difference that I can see is that sparse bundles store the data in bands, tightly compressed, which is why they are vulnerable. Snapshots are also compressed, but not stored in bands, so an error should affect only that file, not the entire bundle, if I understand it correctly.

 

[Slydude]

rbpeirce Thanks for the clarification. I was hoping that there was an error message that might point us in the right direction toward a fix.

Jake, thanks for that quick summary. I knew there were some differences but haven't taken the time to delve into the differences and what they mean on a day-to-day practical basis.

I was just at the Electric Light site to see if there were any updates on whether Disk Utility can repair sparsebundle files.I've used that method in the past but I doubt it would help in this case.

 

[MacInWin]

Sly, this is an older article, but I don't think anything has changed:

Sparse bundles: what they are and how to work around their bugs

An efficient means of storage with many uses, but riddled with bugs and shortcomings. Explains how to compact and resize them, and more.

eclecticlight.co

In it, he says

I bring one disappointment, though. In this article, I wondered whether the sparse bundle format used error-correcting code to improve resilience against corruption. Sadly, I can find no evidence of that from looking through the storage bands inside sparse bundles. Instead, they contain a lot of unused space, frequent repetition of similar data structures which might even amount to redundancy, even a hidden EFI volume, which enable them to withstand substantial amounts of corruption across much of their length. But if any of the file system structures within the bands are damaged, the sparse bundle will be irrevocably broken. They’re not smart, just lucky.

That tells me that sparse bundles have no error correction capability internally. And that is what makes them fragile.

 

[Slydude]

Thanks, I didn't see that article when I scanned through the site a few minutes ago. It has been a while since I last visited and the amount of information available has exploded. The last few times I've been there it was for a specific article that someone recommended.

 

[MacInWin]

It is becoming my go-to site for details on the workings of macOS. His detailed research is well worth reading and staying up to date.

 

[Slydude]

In the times that I have been there before much of the information covered something I knew (which was covered in more detail) or wasn't relevant for me at the time. Now that I'm making the transition to Apple Silicon and Ventura I've got some catching up to do.

 

[rbpeirce]

I do have The Time Machine Mechanic. The question is on which machine do I want to run it? Time Machine runs on my wife's computer but the sparsebundle is on mine. When I run it on my machine I'm pretty sure it is checking my Time Machine backups, but I don't have it installed on hers.

 

[MacInWin]

I also use a lot of his utilities. In addition to The Time Machine Monitor (T2M2), I use Silentknight, XProCHeck, xattred, Ulbow, Sparsity, Mints, and a couple of others. T2M2 is great for seeing what TM is doing, Ulbow allows a deeper exploration of the Unified Log, Mints has lots of ways to get at particular parts of the Unified Log.

For example, Apple just released updates to XProtect and XProtect Remediator. I could just wait, but I use Silentknight to do the update for me.

 

[MacInWin]

This is what I said.

Use it to see what is going on in the logs on both machines.

 

[rbpeirce]

I decided I needed to run it on my wife's machine. It said no backups were completed successfully in the last eight hours and there were no errors. I guess failure to complete a backup, which is the problem I'm trying to solve, is not an error!

 

[rbpeirce]

Try getting T2M2 from here:

XProCheck, T2M2, LogUI, Ulbow and log utilities

XProCheck – check on XProtect Remediator scans completed and reported in the log Updated! Inspects the Unified log for a period of 1-30 days and reports all completed and reported anti-malwar…

eclecticlight.co

Use it to see what is going on in the logs on both machines.
SLY, there are differences between sparse bundles and snapshots. If you do a search at Eclecticlight.com, there are dozens of articles on each. The biggest difference that I can see is that sparse bundles store the data in bands, tightly compressed, which is why they are vulnerable. Snapshots are also compressed, but not stored in bands, so an error should affect only that file, not the entire bundle, if I understand it correctly.

Vulnerable. What does that mean? Vulnerable to corruption is one thing; I can fix that by restoring a previous good backup file. What about vulnerability to not completing properly? that's the problem I'm having.

 

[MacInWin]

I decided I needed to run it on my wife's machine. It said no backups were completed successfully in the last eight hours and there were no errors. I guess failure to complete a backup, which is the problem I'm trying to solve, is not an error!

Go further back. It sounds like the mechanism that triggers a backup on her machine isn't doing that. If you search the entire log, you MAY find it, but if the log doesn't go back to when the problem started (it gets pruned to conserve disk storage), then she can reboot to see if that triggers the scheduler again.

Bottom line is that failing to complete a backup is NOT an error if no backup was started.

 

[MacInWin]

Vulnerable. What does that mean? Vulnerable to corruption is one thing; I can fix that by restoring a previous good backup file. What about vulnerability to not completing properly? that's the problem I'm having.

Vulnerable means that there is no error correction built into sparse bundles. No way for the bundle to detect that a bit has switched from 1 to 0 or vice versa. No way to rebuild the broken part. Restoring from a previous good file is equally vulnerable as there is no error correction code in that previous good file, either. Not even any guarantee the previous "good" file was actually "good" at all.

With normal files, there are various error correcting codes built in so that if something gets wrongly saved, it can be recovered. Think of checksums, for example, although that is kind of brute force ECC. If a block of data has an error, the checksum won't match, the block is corrupted. Sophisticated ECC can actually calculate where the error is and repair it. You never know the error was there.

But no such mechanism exists in sparse bundles. Hence, vulnerable.