mirror of
https://github.com/trapexit/mergerfs.git
synced 2026-10-19 16:16:48 +00:00
Update docs on limiting drive spinup
This commit is contained in:
committed by
trapexit
parent
6d218fa87f
commit
9e2e1afafa
@@ -45,9 +45,7 @@ so it'd have to be on a SSD or other storage device. If anything
|
||||
changed out of band it would break things in weird ways (and MANY
|
||||
users depend on out of band changes working.) It could rescan on
|
||||
occasion but that would require spinning up everything. Filesystem
|
||||
watches could be used to get updates when the filesystem changes but
|
||||
that would allow for subtle race conditions and might keep the drives
|
||||
from spinning down. Also, since mergerfs is just another piece of
|
||||
watches could be used but since mergerfs is just another piece of
|
||||
software interacting with the filesystem ALL mergerfs requests would
|
||||
be echoed back to it causing lots of overhead throwing away a super
|
||||
majority of events it triggered. And something as "simple" as keeping
|
||||
@@ -57,11 +55,11 @@ technologies distort the meaning of usage values.
|
||||
|
||||
Only if ALL metadata and all file locations were cached could mergerfs
|
||||
then do very targeted file operations so as not to trigger a scan
|
||||
across the pool. But that would break out of band interactions. It
|
||||
across the pool. But that will break out of band interactions. It
|
||||
would require accepting possibly bad behavior by space based policies
|
||||
since space would be imprecisely computed. It would require long scans
|
||||
of the filesystems on startup. It would require fundamental changes to
|
||||
how mergerfs works.
|
||||
of the filesystems on startup and a separate location to store all
|
||||
this data.
|
||||
|
||||
|
||||
### What if you only query already active drives?
|
||||
@@ -80,17 +78,77 @@ Yes, but whether it works for you depends on your tolerance for the
|
||||
complexity.
|
||||
|
||||
1. Cleanly separate writing, storing, and consuming the data.
|
||||
1. Use a SSD or dedicated and limited pool of drives for downloads / torrents.
|
||||
2. When downloaded move the files to the primary storage pool.
|
||||
3. When setting up software like Plex, Jellyfin, etc. point to the
|
||||
1. Use a SSD or dedicated and limited pool of drives for downloads / torrents.
|
||||
2. When downloaded move the files to the primary storage pool.
|
||||
3. When setting up software like Plex, Jellyfin, etc. point to the
|
||||
underlying filesystems. Not mergerfs.
|
||||
2. Add a bunch of bcache, lvmcache, dm-cache, or similar block level
|
||||
cache to your setup. After a bit of use, assuming sufficient
|
||||
storage space, you can limit the likelihood of the underlying
|
||||
spinning disks from needing to be hit.
|
||||
3. Tweak your system's settings around caching:
|
||||
* mount all your filesystems with `noatime` to disable updating
|
||||
"last accessed" time on files.
|
||||
* mount all your filesystems with `nodiratime` to disable updating
|
||||
"last accessed" time on directories.
|
||||
* `sudo sysctl -w vm.vfs_cache_pressure=10`: Prioritizes keeping
|
||||
file metadata cached in RAM.
|
||||
* `sudo sysctl -w vm.dirty_ratio=40`: Allows up to 40% of system
|
||||
memory to be filled with unwritten "dirty" data before forcing a
|
||||
disk flush.
|
||||
* `sudo sysctl -w vm.dirty_background_ratio=5`: Sets a lower
|
||||
threshold (5%) for the kernel to start writing data in the
|
||||
background without blocking applications.
|
||||
* `sudo sysctl -w vm.dirty_writeback_centisecs=6000`: Increases
|
||||
the interval between kernel writeback checks to 60 seconds
|
||||
(default is usually 5).
|
||||
* `sudo sysctl -w vm.dirty_expire_centisecs=12000`: Allows data to
|
||||
stay "dirty" in RAM for up to 120 seconds before it must be
|
||||
written to disk.
|
||||
* To make `sysctl` values permanent add the above to
|
||||
`/etc/sysctl.conf`.
|
||||
* **Note:** for systems with limited RAM the above `sysctl` values
|
||||
could cause issues.
|
||||
* **Note:** if you increase the time for which data is stored in
|
||||
RAM before being flushed to disk there is also an increased risk
|
||||
that a sudden powerloss could lead to lost data.
|
||||
4. Enable all the appropriate caches in mergerfs:
|
||||
* [https://trapexit.github.io/mergerfs/latest/config/cache/](https://trapexit.github.io/mergerfs/latest/config/cache/)
|
||||
* `cache.entry=86400`: cache file existence for 24h.
|
||||
* `cache.negative-entry=86400`: cache file non-existence for 24h.
|
||||
* `cache.attr=86400`: cache file metadata for 24h.
|
||||
* `cache.symlinks=true`: enable symlink caching.
|
||||
* `cache.readdir=true`: enable directory listing caching.
|
||||
|
||||
Remember too that while it may be a trade off you are willing to live
|
||||
with there is decent evidence that spinning down drives puts increased
|
||||
wear on them and can lead to their death earlier than otherwise. You
|
||||
may end up saving money on electricity but spending more on having to
|
||||
purchase new drives.
|
||||
|
||||
### References
|
||||
|
||||
* [Documentation for /proc/sys/vm/](https://docs.kernel.org/admin-guide/sysctl/vm.html)
|
||||
* [How to Configure vm.swappiness and Other Virtual Memory Parameters on RHEL](https://oneuptime.com/blog/post/2026-03-04-configure-vm-swappiness-virtual-memory-rhel-9/view)
|
||||
|
||||
|
||||
### Tradeoffs
|
||||
|
||||
While it may be a trade off you are willing to live with there is
|
||||
decent evidence that spinning down drives puts material increased wear
|
||||
on them and can lead to their death earlier than otherwise. You may
|
||||
end up saving money on electricity but spending more on having to
|
||||
purchase new drives sooner.
|
||||
|
||||
|
||||
## Alternatives
|
||||
|
||||
[Project Comparisons](../project_comparisons.md) lists a few projects
|
||||
that are similar to mergerfs but offer caching. Please note that these
|
||||
projects are very young and not general purpose like mergerfs. Their
|
||||
compliance to POSIX filesystem rules is probably low enough that
|
||||
certain behaviors may not work or worse lead to data corruption.
|
||||
|
||||
I'm not suggesting not to look at those projects but filesystem
|
||||
development is complicated even when it appears to be just a layer on
|
||||
top of another. Just be careful and do your research. That stands for
|
||||
anything touching your data... mergerfs included.
|
||||
|
||||
To the degree that mergerfs can add features which help in this
|
||||
situation, so long as it doesn't interfere with other fundamental
|
||||
features, they may be added over time.
|
||||
|
||||
Reference in New Issue
Block a user