Available Disk Space Constantly in Flux w/ APFS, High Sierra

[lehestro]

Hello all. I've looked around at various forums n such. There's been some mention of this, but I've never been able to get a clear, definitive take on what's happening.

I'm using High Sierra, and because of what I'm assuming is some peculiarities with APFS, my available disk space is constantly in flux. Often this isn't a problem, but sometimes it is. For example, I need massive amounts of disk space allocated for the cache folders of certain video software, and I get error messages (particularly if it's right when I boot up the computer) that says I do not have enough disk space.

The amount of space Finder reported available today ranged from 4GB-138GB. Omnidisksweeper seems confused as well. I often go to the menu bar Apple Icon>About This Mac>Storage and this seems to get my file system to recalculate the available space.

My question is: what exactly is going on? Why is my available space seemingly always in flux? It should be noted that I do not use iCloud drive, and I have "optimize storage" in the iCloud system settings turned to 'off'.

I used to be assiduous in managing my disk space availability, but now it's chaos. Many thanks to anyone that can break this down properly for me.

 

[Raz0rEdge]

What Mac? How much total storage? How much RAM? How much is used? APFS shouldn't randomly report varying free space unless something is taking up the space and releasing it. That sort of fluctuating behavior is more reminiscent of RAM than storage.

 

[MacInWin]

The amount of free space on the boot drive is always in flux if the total memory (RAM) of the system is insufficient. What happens is that when the OS demands more memory than is available to perform some task, idle applications are paged out to the boot drive (cached) so that the task can complete. If you have a lot of things going on, and if the memory is tight, that swapping in and out will cause fluctuations in the amount of space available on the drive. Given the small free space you say you have (4GB to 13GB) it's not surprising that the fluctuations seem large. In addition, the applications themselves may be using what is known as "scratch" space for storage of temporary data. That temporary data may be needed for "history" functions in applications that support undo functions. And the deeper the undo is capable of going, the more space is required for that undo process. Again, as things are being edited and saved, that scratch space fluctuates in size.

Bottom line, unless your drive is tiny, 4GB to 13GB is not sufficient room for the functions needed. The general recommendation is for 15-20% free on the boot drive. Assuming the smallest margin of 15%, and the largest room you say you have at 13GB, the math works out that unless your drive is less than 100GB, that's way too little free space, regardless of the format.

 

[lehestro]

Thanks for the response, I appreciate it, but you misunderstand. I'm aware of the need for disk caching/memory swapping, but this is something else entirely. I have a 500GB hard drive that has about 350GB of data on it. Despite that, often times, without any major accruing or deleting of data, my available disk space will read as anything from 5GB to the proper 150GB available. These are HUGE fluctuations – way more than memory swapping would account for (as it stands I have 64GB of memory, so not much swapping occurs anyhow). These fluctuations occur at random, and change very quickly – it'll report 40GB of free space, and 2 minutes later 120GB of free space, and then will go back down again.

I know I can't be the only person experiencing this, but haven't really been able to find much about it.

 

[lehestro]

What Mac? How much total storage? How much RAM? How much is used? APFS shouldn't randomly report varying free space unless something is taking up the space and releasing it. That sort of fluctuating behavior is more reminiscent of RAM than storage.

It's a 2013 mac pro (trash can). I have 64GB of Ram, of which only about 40GB is ever used. I have had this computer from Lion (I think – maybe Mountain Lion) all the way to High Sierra (skipping Sierra). Never had this issue until now.

I saw somewhere that turning on "optimize storage" in iCloud drive and/or System Report>Storage>Manage could have this be the case, what with Apple's new systems to have the OS manage storage for you, but I have those things turned off, best to my knowledge.

It's more akin to some sort of constant indexing or something. I don't know. I'm a power user, but no computer science/programmer. This is beyond me.

 

[MacInWin]

You said in post #1:

For example, I need massive amounts of disk space allocated for the cache folders of certain video software, and I get error messages (particularly if it's right when I boot up the computer) that says I do not have enough disk space.

That is the pointer to your problem. As you are running applications (whatever video editor you are using) the amount of scratch or cache space needed will fluctuate. When the editor opens the file, it creates the scratch space the size of the video (Let's say 5GB). Then, depending on how many layers of "undo" you have set, a new copy is created for each undo layer. If there are 10 undo steps set, then another 50GB will be needed for those undo versions. Finally, one more copy is needed for the current copy being edited. So with 10 undo of a 5gB file you need 60GB of space. When you do the final save of that video file, all 60GB is returned to free space. But let's say you want to merge two videos. Now you have two 5GB video originals, plus two sets of 10 undo versions, plus 2 current working copies, plus the new 10 GB version you are creating, making the total demand 130GB. And as soon as you save that project, it all comes back to you as free again.

This is all theoretical, as you haven't given any names of the video software, but is the model that Photoshop uses, and I think iMovie used the same approach. One way to cut down on the space demanded is to reduce the undo history depth (if you can), but then you give up the ability to undo as far back.

Or get a bigger drive. These days 500GB is pretty small.

One final thing. You said, "I often go to the menu bar Apple Icon>About This Mac>Storage and this seems to get my file system to recalculate the available space." That is a notoriously inaccurate report of storage used and free. I don't even know why Apple included it, it's so bad.

 

[harryb2448]

If you are running a slow old platter drive, the best bang for your buck is get an SSD and install it, use the platter in Bay 2 as your backup drive.

 

[lehestro]

You said in post #1:That is the pointer to your problem. As you are running applications (whatever video editor you are using) the amount of scratch or cache space needed will fluctuate. When the editor opens the file, it creates the scratch space the size of the video (Let's say 5GB). Then, depending on how many layers of "undo" you have set, a new copy is created for each undo layer. If there are 10 undo steps set, then another 50GB will be needed for those undo versions. Finally, one more copy is needed for the current copy being edited. So with 10 undo of a 5gB file you need 60GB of space. When you do the final save of that video file, all 60GB is returned to free space. But let's say you want to merge two videos. Now you have two 5GB video originals, plus two sets of 10 undo versions, plus 2 current working copies, plus the new 10 GB version you are creating, making the total demand 130GB. And as soon as you save that project, it all comes back to you as free again.

This is all theoretical, as you haven't given any names of the video software, but is the model that Photoshop uses, and I think iMovie used the same approach. One way to cut down on the space demanded is to reduce the undo history depth (if you can), but then you give up the ability to undo as far back.

Or get a bigger drive. These days 500GB is pretty small.

One final thing. You said, "I often go to the menu bar Apple Icon>About This Mac>Storage and this seems to get my file system to recalculate the available space." That is a notoriously inaccurate report of storage used and free. I don't even know why Apple included it, it's so bad.

Unfortunately I can't get a bigger drive. The flash drives on the 2013 Mac Pros are insanely expensive, especially considering it'd be for beefing up 4-5 year old tech. (I think a TB is $1k? Or $800 or something). And while I see what you're saying, and I think you pointed to some things that I wasn't entirely considering, the fact remains that I have 46GB alloted for media cache files. This is for premiere and after effects. What happens with this system (at least, historically) is as that cache allotment fills up, older cache files are deleted, so you never go past your maximum. I'm used to this sort of fluctuation on my hard disk. With that 46GB full, I still should have ~120GB free. And sometimes I do. That's the weird thing.

I've had this computer for many years, and have been using these programs for longer. The crux is, this was never ever an issue until High Sierra and APFS (I skipped Sierra). Something is afoot. I've found some other threads have mentioned a similar sort of problem.

Another thing to note: if everything could be attributed to what you're talking about, then there should be no flux as I sit still, and stop working. However, when I do just that, if I'm at something really low (say 11GB), I can slowly watch the available space creep back available. There should be no reason for this, except the OS is up to something; this isn't something Adobe's programs. This happens, at first, about 300MB at a time, but seems to increase exponentially, not linearly. I've attached a screen cap of this.

 

[lehestro]

I guess I'm having trouble getting my screen cap video to show up in my post. Here's a vimeo link: http://vimeo.com/252635382

It takes about until the 11s mark for there to be a change. This is me with all programs closed, sitting idle staring at my finder window.