Compare commits

...
173 Commits
Author SHA1 Message Date
Antonio SJ Musumeci 8e87bcbff2 docs: Add Unraid migration guidance for mergerfs
- Add a top-level Migration page covering both migration directions.
- Explain filesystem reuse, parity constraints, partition requirements,
  and unlocking encrypted Unraid disks.
- Describe intact ZFS pool imports and subdirectory branch reuse,
  distinguishing them from cases requiring a data copy.
- Add navigation and link the guide from the Unraid comparison.
2026-10-03 16:52:25 -05:00
Antonio SJ Musumeci fcd7527c9d docs: document mergerfs tradeoffs and web UI management
* Add: a comprehensive downsides FAQ with mitigations and practical
  priorities.
* Document: mergerfs-webui and link it from setup and configuration
  guides.
* Replace: obsolete mergerfs.fsck references with fsck.mergerfs
  guidance.
2026-10-01 20:14:52 -04:00
Antonio SJ Musumeci 83d1630f55 docs: rework and expand project comparisons 2026-09-22 14:50:47 -05:00
Antonio SJ Musumeci cfb925e5ae docs: clarify branch resolution and mount ordering 2026-09-21 13:46:38 -05:00
Antonio SJ Musumeci 515b0549ef docs/performance: explain mergerfs IO performance vs RAID
Describe why mergerfs does not aggregate branch performance like
RAID/ZFS (whose aggregate throughput is also capped by shared
interconnects) and instead provides independent per-branch
performance: best case A+B+C across concurrent streams, worst case
single-stream limited to one branch's speed.
2026-09-05 16:00:43 -04:00
natalefalco 8ea47df13e Include Docker Compose setup for mergerfs
Added Docker Compose configuration for running mergerfs.
2026-07-31 08:13:59 -05:00
shyney7 42ab831d97 docs/setup: link install :z example to SELinux FAQ 2026-05-17 10:14:21 -05:00
shyney7 89b4046d11 docs/faq: add SELinux relabeling entry 2026-05-17 10:14:21 -05:00
Antonio SJ Musumeci 7ec6f8e2f2 Manage cloned /dev/fuse fds in sessions
- Store cloned read-worker fds on fuse_session and close them there.
- Set up clone fallback through session helpers used by the loop.
- Carry per-request fds through receive, process, and reply paths.

Tests show significant performance increases in highly concurrent
workloads on a high core count system. Upto 15% better on reads and
27% on writes.
2026-05-14 20:13:07 -05:00
Antonio SJ Musumeci 5f7fa17ad8 Inline ioprio disabled fast path
- Expose the ioprio enabled flag as an atomic for relaxed inline checks
- Inline SetFrom construction so disabled read/write paths avoid a function call
- Move priority application into a slow helper used only when ioprio is enabled
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 30d59d4458 Use shared_mutex for fuse log config
- Guard fuse_cfg_t log file and filepath shared pointers with a shared mutex
- Replace atomic shared pointer loads and stores with shared and unique locks
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 02dc36080c Use shared_mutex for ReadDir state
- Replace atomic shared pointer loads/stores with shared/unique locking.
- Store the configured readdir mode directly as std::string.
- Copy the active ReadDir implementation under lock before invoking it.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 8734847968 Guard branch snapshots with shared_mutex
Important changes:
- Replace atomic shared_ptr operations with shared_mutex-protected snapshots.
- Retry copy-on-write branch updates when another writer publishes first.
- Keep long-running parsing and read-only filesystem checks outside branch locks.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 608cd00764 Use atomic open file reference counts
- Use relaxed increment under the map bucket lock when duplicating file handles.
- Use acq_rel decrement when releasing and checking for final references.
- Load moved ref_count with relaxed ordering during map insertion.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci b13727abde Document readdir merge mutex scope
- Explain that the mutex is per readdir call.
- Document that only dirent deduplication and list append are serialized.
- Clarify that per-branch opendir and getdents64 work runs outside the critical section.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 231aac15cd Use shared write locks until ENOSPC relocation
- Replace the per-file Mutex with std::shared_mutex for shared write access.
- Run direct and cached writes under a shared lock while the fd is stable.
- Retry failed writes under a unique lock before move-on-ENOSPC relocation.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 8b17d5f35e Synchronize node updates with path release
- Add helpers that release path locks and update node state under f.lock.
- Update getattr and setattr stat-cache invalidation while preserving ESTALE handling.
- Rework create, open, tmpfile, and set_path_info to mutate lookup/open state under the shared lock.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 8608b2be46 Clarify releasedir readdir synchronization barrier
- Document the lock/unlock pair as an acquire/release barrier.
- Explain that releasedir waits for in-flight readdir calls before destroy/free.
- Warn against removing the synchronization despite it not being a critical section.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 1e0bd54989 Move statvfs cache to use shared mutex
Its read heavy and micro benchmarks show no regression on the low end
and 30%+ throughput improvements on the highend.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 56285fc6e5 Pass annobin override to RPM toolset builds
Important changes:
- Add Makefile RPMBUILD and RPMBUILD_FLAGS overrides for the rpm target.
- Detect /opt/rh GCC toolsets before RPM packaging.
- Pass --undefine _annotated_build to avoid Rocky 8 annobin plugin failures.
2026-05-13 17:01:41 -04:00
Antonio SJ Musumeci dfe053a0ae Require C++20-capable build toolchains
Important changes:
- Switch mergerfs and vendored libfuse Makefiles to -std=c++20.
- Select the latest available GCC/G++ or toolset in container builds.
- Verify compiler C++20 support during package installation.
- Fix dynamic SysLog formatting for fmt C++20 checks.
2026-05-13 13:55:43 -05:00
Antonio SJ Musumeci 12ac368e8b Preserve CPPFLAGS in release builds
- Append -DNDEBUG to existing CPPFLAGS in the release target.
- Keep caller-provided build flags for Debian packaging.
2026-05-09 19:28:56 -05:00
Antonio SJ Musumeci c8f32f9f25 update build system and dependency configuration 2026-05-09 18:55:23 -05:00
Antonio SJ Musumeci ce3737ff33 Add new OS build files 2026-05-07 23:35:14 -04:00
Antonio SJ Musumeci 84e4cdeb7e make all integration tests self-contained with ephemeral mounts
- Add mergerfs_mount() context manager in posix_parity.py that creates
  temporary branch dirs in tests/.test_tmp/, mounts an ephemeral mergerfs
  instance, and cleans up after use.
- Add find_mergerfs(), find_fusermount(), mount_mergerfs(), unmount_mergerfs()
  helpers.
- Update run-tests to invoke each TEST_* script without a mountpoint arg.
- Update README.md to document the new `python3 tests/run-tests` invocation.
- Fix 44 tests to use mergerfs_mount() instead of expecting a pre-mounted path.
- Add tempfile import to 10 tests that were missing it.
- Fix TEST_mount_lifecycle to use mergerfs_mount() context manager.
- Fix TEST_posix_tmpfile missing join import.
- Fix TEST_cfg_xattr_modes error message framing (no "(known bug)").
- Fix TEST_posix_copy_file_range: loop-copy full size, compare st_size and
  SHA256 hash; add NativeTmp for space management; route native temp to
  tests/.test_tmp/ to avoid ENOSPC on /tmp.
- Fix TEST_posix_getattr_fgetattr: st_cmp skips st_size for dirs; add
  compare_ino param for safe inode comparison when inodecalc=passthrough.
- Fix TEST_unlink_rename, TEST_use_* tests: namespace cleanup, consistent
  variable naming, None check for cmpfunc.
- Fix TEST_readlink_semantics: remove type annotations to support older
  Python; wrap in mergerfs_mount() try/except.
- Add flush=True to run-tests print so test names show immediately.
2026-05-07 21:47:40 -04:00
Antonio SJ Musumeci 6d6d73c7b3 Use unmount() on FreeBSD for ENOTCONN umount 2026-05-07 17:49:24 -04:00
Antonio SJ Musumeci bea31543a0 update fuse_lib_tmpfile() node management to match fuse_lib_create() 2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci 99835a0178 Document advisory locking limitation and update deprecated options
- Add documentation explaining that advisory file locking is not supported

- Add remember-nodes to deprecated options mapping

- Add comment explaining stat_crc32b fingerprint invalidation behavior

- Fix heading level for Host vs Container identity section
2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci 342a1efcfb Add QEMU binfmt setup and fix static installer
- Fix typo in static.installer: /pkg/ -> /pkgs/

- Add missing mkdir -p /build before makeself

- Mount pkgs directory via podman build -v so static.installer can access

  previously built static tarballs without relying on bind mounts relative

  to the build context

- Register qemu-user-static binfmt handlers for cross-compilation

- Make --git-repo optional when using --setup
2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci c587a127fb add missing <new> include for std::nothrow in fuse_session.cpp 2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci f1034a3ad6 Remove timed node removal 2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci 8117998205 Remove unused lock operation plumbing 2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci cc780bc521 libfuse: remove per-node lock bookkeeping 2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci 14b7c8e607 Use stat fingerprint as cache validity sentinel
- Reserve a zero stat CRC to represent an invalid cache fingerprint.

- Include device, size, inode, mtime, and ctime data in the fingerprint.

- Recompute the fingerprint during auto-cache open before keeping cache.
2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci 3ff9794fff libfuse: fix use-after-free in fuse_lib_ioctl 2026-04-30 12:51:12 -05:00
Antonio SJ Musumeci f71e27644e vendor moodycamel queue and upgrade to 1.0.5 2026-04-30 08:43:42 -04:00
Antonio SJ Musumeci 943f91ff98 Strengthen mergerfs integration tests
- Make the test runner fail on failed tests and report explicit skips
- Strengthen assertions for O_DIRECT, LUP policy, EXDEV fallback, and xattrs
- Fix cleanup and file descriptor handling across test scripts
2026-04-29 20:25:20 -04:00
Antonio SJ Musumeci ef1e01b5a5 Add unit and integration tests for recently changed code
Unit tests (29 new in tests/tests.cpp):
- str::eq() and str::startswith() char* nullptr safety
- str::from() u64/s64/int suffix parsing (K/M/G/T, overflow, invalid)
- num::humanize() byte formatting (0, 512, 1K, 1M, 1G, 1T, non-exact)
- RmdirErr priority error accumulation (ENOTEMPTY/EEXIST override rules)
- HashSet inline-to-heap growth, empty string, and large insertion
- fs::inode algorithm switching (valid/invalid names, passthrough raw ino,
  distinct inodes across algorithms)
- RND::rand64() bounds (range, max, shrink_to_rand_elem)
- Branches::Impl copy and move assignment re-links default_minfreespace

Integration tests (5 new in tests/):
- TEST_policy_lup: LUP policy create and search
- TEST_rmdir_enotempty_priority: rmdir returns ENOTEMPTY for non-empty dirs
- TEST_open_after_unlink: read/write via fd after unlink
- TEST_api_xattr: mergerfs xattr API (fullpath, relpath, basepath, allpaths)
- TEST_movefile_enospc: moveonenospc configuration round-trip

Also: remove xattr nosys checks from TEST_cfg_xattr_modes, reformat
integration test scripts to use temp_dir/cleanup_dir helpers, and
minor parameter alignment changes in fuse_getxattr, fuse_init,
fuse_rename.
2026-04-29 17:41:00 -04:00
Antonio SJ Musumeci 50c0627b2b Better handle make sanitize option 2026-04-29 12:24:10 -04:00
Antonio SJ Musumeci 660f556ac6 Don't return success on ENOTEMPTY rmdir errors 2026-04-29 12:23:49 -04:00
Antonio SJ Musumeci 997b036064 Reject invalid policies 2026-04-29 00:34:00 -04:00
Antonio SJ Musumeci 9308a3729c When file unlinked check open_files map
Inadvertantly removed. Didn't realize that the kernel would not always
send file handle when unlinked.
2026-04-28 16:04:26 -04:00
Antonio SJ Musumeci 7616d8b1ac Rework readlink to not depend on cstr nul termination 2026-04-26 01:37:08 -04:00
Antonio SJ Musumeci a86092b3c2 Don't push docs for RC tags 2026-04-23 20:46:55 -05:00
Antonio SJ Musumeci e613d9415f Better same file check for fsck 2026-04-22 22:28:56 -05:00
Antonio SJ Musumeci 1a8eb326af Tweak docs on NFS and identity management 2026-04-21 13:20:07 -05:00
Antonio SJ Musumeci 9fb1642e93 Fix docs regarding "noforget" 2026-04-20 23:25:47 -05:00
Antonio SJ Musumeci 8be86ee611 Remove mkdocs build github workflow 2026-04-19 23:48:13 -05:00
Antonio S. Musumeci da68cf3a85 Formatting: normalize brace style and parameter alignment
No functional change: libfuse/helper.cpp (fuse_daemonize,
fuse_helper_opt_proc, fuse_parse_cmdline) and mergerfs
fuse_init.cpp (_want helpers) reformatted to match the project's
GNU-style brace placement and aligned parameter lists.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci ec40866091 libfuse: small safety and portability fixes across opt/cpu/fusermount/maintenance_thread
- fuse_opt: reformat process_opt_param to project style; return the
  rv from sscanf through a named local so the error branch prints
  the failing option cleanly.

- cpu.cpp: when parsing /sys/devices/system/cpu/*/topology/core_id,
  bail out cleanly on stream read failure rather than recording an
  uninitialized core_id.

- fusermount: free(copy) on the success path of parsing the mount
  option (previously leaked on every mount), and replace the fixed-
  buffer sprintf into procfile with snprintf bounded by the buffer
  size.

- maintenance_thread: widen the iteration count from int to u64
  so a long-lived daemon doesn't wrap after ~68 years of ticks
  (real effect: silence signed-overflow warnings under UBSan).
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci bd0714272e libfuse: thread_pool/objpool/fuse_loop - safer memory and iteration
- ThreadPool::destroy used to iterate _threads directly while another
  thread could be adjusting the same vector. Snapshot the vector
  under the mutex and iterate the local copy for pthread_cancel /
  pthread_join.

- ObjPool::construct now returns NULL when the pooled allocator
  returns NULL instead of invoking placement-new on a null
  pointer. In destroy, compute to_node(obj) before running the
  destructor - the destructor may scrub bytes the to_node helper
  reads.

- In the async/sync workers in fuse_loop, msgbuf_alloc can return
  NULL under memory pressure. Fail the worker cleanly with a
  diagnostic rather than dereferencing. Also introduce a small
  _nproc() helper that treats hardware_concurrency()==0 as 1, so
  the downstream min/max arithmetic doesn't produce a zero thread
  count.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci 27acd9fcc7 libfuse: use atomic fuse_session pointer in signal handler
The signal handler reads the global session pointer from an async-
signal context while another thread installs/removes the session
via set/remove_signal_handlers. Use std::atomic<fuse_session*> with
acquire/release ordering so the handler either sees NULL or a fully
published session, never a torn pointer.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci 4516685ca7 libfuse: harden fuse.cpp against NULL nodes; use lock guards; check allocations
Collapse get_node/get_node_nocheck into a single function that
returns NULL when the node is absent rather than abort()ing.
Callers that previously assumed non-NULL now check the return value
and either reply ESTALE/ENOMEM or fall through gracefully; a few
places where the node is load-bearing (e.g. unhash during remove)
use fatal::abort with a descriptive message.

Switch mutex_lock / mutex_unlock pairs to mutex_lockguard so early
returns on error paths no longer leak the lock, and add NULL-safety
around fuse_dh in readdir/readdir_plus, fuse_dirents_init error
returns, the getattr path when both name and fh are empty, and the
msgbuf allocation in fuse_lib_read.

Pull the new xstrdup/xmalloc wrappers from xalloc.hpp for the root
node name so the common-case OOM path terminates with a diagnostic
instead of a silent corrupted node.

Also guard fuse_session_destroy against a NULL session - called
from error-recovery paths.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci 0bdbaf1d6c libfuse: remove userdata from init/destroy APIs; drop unused fuse_reply_ioctl_iov
The low-level init/destroy callbacks were passed a userdata pointer
that mergerfs never consumed - the caller threaded it through
fuse_lowlevel_new only to hand it back unchanged. Remove the
parameter from the ops struct, from fuse_lowlevel_new(_common),
and from the internal fuse_ll struct; update fuse_lib_init and
fuse_lib_destroy accordingly.

Also drop fuse_reply_ioctl_iov: no caller in-tree invokes it and
the public API is defined by this vendored libfuse fork.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci e67b7d0825 log: default debug log path to empty; skip set when empty
libfuse previously initialized its debug log filepath to
"/dev/stderr", forcing the config layer to reopen stderr on every
load whether or not the user asked for it. Default the libfuse log
path to "" and have the mergerfs config layer only forward the
request to libfuse when the user supplied an explicit path.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci 981d5473ee str: null-safe same() and startswith()
Both helpers previously called strcmp/pointer-dereference directly
on their arguments, so a stray NULL (e.g. an unset env string or an
absent xattr) crashed the process. Guard both helpers: same()
returns true only if both pointers are equal, startswith() returns
false if either side is NULL.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci 364f8c4407 fs_movefile_and_open: -ENOSPC when no destination branch available
The create policy can legitimately return success with an empty
destination list (e.g. every candidate rejected). Previously we
then dereferenced dst_branch[0] and faulted. Return -ENOSPC so the
original write that triggered the move propagates a sensible errno
to userspace.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci ef9c7ad5df fuse_statfs: scale with long double to avoid integer overflow
The branch-aggregation code computed
  blocks * f_frsize / min_frsize
as u64 arithmetic. With large filesystems the intermediate product
blocks * f_frsize overflows 64 bits long before the division can
bring the result back into range, producing garbage bavail/bfree
values reported to userspace.

Compute the scale factor once in long double and multiply each
count by it, so the only narrowing happens in the final cast back
to fsblkcnt_t.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci 4288a60c5d fuse_readlink: always null-terminate buffer
Old code only null-terminated when the returned/copied length was
strictly less than the caller-supplied bufsize_, leaving the buffer
unterminated on an exact-fill. Since buf_ is consumed as a C string,
reserve one byte for the terminator up front and unconditionally
terminate.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci cf38a37176 fuse_readdir: atomic initialized flag; bound error string; check msgbuf allocation
- ReadDir::_initialized is set by the first thread through the
  lazy-init path and read by every subsequent caller; make it
  std::atomic<bool> so the read/write pair has a happens-before
  relationship.

- The 'no valid mergerfs branch found' error directory entry used
  strcpy into a fixed-size dirent buffer. The message happens to
  fit today but the buffer is only NAME_MAX wide; switch to a
  length-bounded strncpy driven by the constexpr string size.

- The getdents implementations for seq/cor/cosr allocated a msgbuf
  without checking for allocation failure and then dereferenced it.
  Return -ENOMEM when the allocation fails.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci f5d9a60b9d fuse_getattr/statx: populate timeouts in fake_root and controlfile replies
The fake-root and control-file fast paths returned a synthetic stat
without touching the timeout struct. Callers that took those paths
ended up sending whatever the timeout struct happened to hold,
which is uninitialized memory for the control-file path and
whatever a prior call left there for the fake-root path. Kernel
behavior is undefined when the timeout is garbage.

Pass timeout_ into both helpers and set entry/attr from the
configured cache values.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci 0c17492c78 FUSE: pass Branches::Ptr by value with empty-branches guards
Previously most FUSE ops took a reference to the live cfg.branches
object. With atomic_load now producing a shared_ptr snapshot, pass
it by value: the op works against a single stable view of the
configuration for its whole lifetime, immune to concurrent
reconfiguration.

Also bail out with -ENOENT when the policy returns an empty branch
set rather than dereferencing branches[0]. This was reachable if a
policy rejected every branch (e.g. everything readonly while
looking for a writable target).

Covers access, chmod, chown, create, getattr, getxattr, ioctl,
link, listxattr, mkdir, mknod, open, readdir (cor/cosr),
readlink, removexattr, rename, rmdir, setxattr, statfs, statx,
symlink, truncate, unlink, utimens.

While here, in _getattr only dereference st_->st_mode when lstat
succeeded (it is otherwise uninitialized).
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci 98648ae44b FUSE: require valid file/dir handle; drop open_files fallback
Several FUSE ops fell back to looking up a FileInfo via
state.open_files when the kernel-provided fh was zero, returning
-ENOENT if the lookup failed. This path papered over handle-tracking
bugs and added cost on every call. Use the handle directly and
return -EBADF when it is invalid - the kernel should always hand us
a valid fh for an open resource.

Also simplify FileInfo::from_fh / DirInfo::from_fh and their to_fh
helpers: callers now handle the null-pointer case explicitly so the
ASSERT_NOT_NULL checks are redundant.

Covers fallocate, fchmod, fchown, fgetattr, flock, ftruncate,
futimens, read, release, releasedir, write, statx, ioctl (file +
dir), readdir (sequential, concurrent-on-read,
concurrent-on-search), and the passthrough fast paths in
create/open.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci 0dcca0f1a3 branches: re-link default_minfreespace pointer on copy/move; atomically publish branch mode changes
Branches using the default minfreespace held a reference to their
owning Impl's _default_minfreespace. After a copy or move the
pointer still referenced the source Impl, which could become
invalid or diverge after further configuration changes. Switch to
a pointer and re-link it to the destination Impl on copy/move.

While here, convert set_branch_mode to do an atomic_store of a
newly constructed Impl rather than mutating the shared pointer
target in place. Readers holding an atomic_load snapshot are no
longer affected by a concurrent mode change.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci 103cd358f2 state: atomic ref_count in OpenFile
ref_count is read and written from multiple FUSE worker threads
during open/release. Make it std::atomic<int> and provide a move
constructor so OpenFile remains movable when the underlying atomic
is non-movable.
2026-04-18 13:47:23 -05:00
Antonio S. Musumeci aa46001923 Remove unused rwlock.hpp header and includes
rwlock.hpp was left over after migrating to atomic shared pointers
for branches configuration. No code references ReadGuard/WriteGuard
any longer, so drop the header and its stale includes.
2026-04-18 13:47:23 -05:00
Antonio SJ Musumeci ff48ff6047 Handle/ignore SIGHUP 2026-04-17 11:14:05 -05:00
Antonio SJ Musumeci d5a2187713 Fix make race condition 2026-04-16 18:20:00 -05:00
Antonio S. Musumeci 351adc2b20 Add tests for HashSet, rapidhash withSeed consistency, and ReaddirCalc parity with calc 2026-04-12 23:44:23 -05:00
Antonio S. Musumeci a1e681f36b Use ReaddirCalc in all readdir implementations
* avoids per-entry path construction
2026-04-12 23:44:23 -05:00
Antonio S. Musumeci e3a467de80 Refactor fs_inode
* add Algo enum
* ReaddirCalc class
* string_view params
* fs::path overloads
2026-04-12 23:44:23 -05:00
Antonio S. Musumeci cdadda5eb9 Replace khash with custom hash set 2026-04-12 23:44:23 -05:00
Antonio SJ Musumeci 4c9a9d7e15 Upgrade rapidhash 2026-04-12 23:44:23 -05:00
Antonio SJ Musumeci 425e30b50f Use nanosleep rather than yield and return EINTR 2026-04-11 11:30:57 -05:00
Antonio SJ Musumeci 3905355522 Detect OS for release-sample make target 2026-04-08 20:48:02 -05:00
Antonio SJ Musumeci 9e2e1afafa Update docs on limiting drive spinup 2026-04-06 14:01:20 -05:00
Antonio SJ Musumeci 6d218fa87f Misc updates and fixes 2026-04-04 02:46:24 -05:00
Antonio SJ Musumeci 3b10406f95 Add more details to project comparisons 2026-04-04 02:17:06 -05:00
Antonio SJ Musumeci f2c8e5d9eb Add FUSE request_timeout
Not yet hooked up to config parser to set as need to evaluate usage.
2026-04-01 23:20:05 -05:00
Antonio SJ Musumeci b581c5d5e7 Add support for FUSE_COPY_FILE_RANGE_64 2026-04-01 23:20:05 -05:00
Antonio SJ Musumeci 1ebc8cfa5f Enable statx with file handle 2026-03-31 19:53:17 -04:00
Antonio SJ Musumeci acfa797503 Misc formatting changes 2026-03-31 19:53:17 -04:00
Antonio SJ Musumeci 7fc20457ca Rework open file resource management 2026-03-31 19:53:17 -04:00
Antonio SJ Musumeci ecee84b323 Improve proj comparisons + new SnapRAID details 2026-03-31 19:51:28 -04:00
Antonio SJ Musumeci b8776d205d Update docs; add nofs, policyfs and greyhole 2026-03-27 22:54:39 -05:00
Antonio SJ Musumeci 06433a0548 Fallback to fusermount3 where available
Also general cleanup of mount_generic.
2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci f4b602340b Add more integration tests 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci a78aef36e0 Additional check for ENOTCONN on mount 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 0d05523dfd Move maintainence thread func take u64 rather than int
signed rollover is UB
2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 0cfab60f37 replace raw C-strings with std::string in converted libfuse files
util/mount.mergerfs.cpp:
- Remove xstrdup/xrealloc wrappers entirely
- Convert type, source, options, command, setuid to std::string
- Replace add_arg() pointer-rewriting with shell_quote() returning std::string
- Replace add_option() with simple std::string += concatenation
- Fix const_cast UB on string literal assignment

lib/helper.cpp:
- Convert add_default_subtype local subtype_opt: malloc+snprintf+free -> std::string construction

lib/mount_generic.h:
- Convert fuse_mount_sys locals source/type: malloc+strcpy+strcat -> std::string
- Remove 4 unnecessary free() calls (RAII cleanup)
- .c_str() at mount() and fuse_mnt_add_mount() boundaries

lib/mount_util.h:
- Rewrite fuse_mnt_resolve_path internals using std::string: rfind/substr
  instead of manual pointer arithmetic, strrchr, strlen chains
2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 21966ade97 Convert vendored/libfuse C sources to C++17
Convert all C source files in vendored/libfuse/ to C++17:
- lib/crc32b.c    -> crc32b.cpp  (const void* cast)
- lib/fuse_opt.c -> fuse_opt.cpp (designated init, void* arithmetic)
- lib/fuse_signals.c -> fuse_signals.cpp
- lib/helper.c   -> helper.cpp  (void* data cast)
- lib/mount.c    -> mount.cpp  (thin wrapper)
- lib/mount_generic.c -> mount_generic.h (static inline, #pragma once)
- lib/mount_bsd.c -> mount_bsd.h    (static inline, #pragma once)
- lib/mount_util.c -> mount_util.h   (static inline, #pragma once)
- util/fusermount.c -> fusermount.cpp
- util/mount.mergerfs.c -> mount.mergerfs.cpp

Key changes:
- Replace C99 designated initializer in fuse_opt_parse with field-by-field init
- Fix void* pointer arithmetic via char* base + offset pattern
- All C-style (T*)voidptr casts replaced with static_cast/reinterpret_cast
- Remove redundant #define _GNU_SOURCE (already in FUSE_FLAGS)
- Wrap __clone2 declaration in extern "C" for ia64 path
- Update Makefile: empty SRC_C, build utils with $(CXX) instead of $(CC)
- Remove stale .c files now replaced by .cpp/.h counterparts

All binaries build cleanly: mergerfs, mergerfs-fusermount, mount.mergerfs
2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 96eb51eecb Move fuse_session to C++ and make exit flag atomic 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 881147daa2 Fix null ptr handling issue in ioctl processing 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 0f1a8e9b69 Fix error condition in fs_copyfile 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 08f9cc6932 Fix readlink implementation, add more tests 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 3099b15db1 More aggressive error handing with pagesize queries 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 07b1e6bdba Auto-commit fuse_msgbuf.cpp 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 7763c7384a Use fatal::abort() rather than assert in places 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 61365d226d Move scope_guard to vendored/ 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci f33e78bc24 Cleanup procfs and abort on critical issues 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 7f52623536 Cleanup branches and add more unit tests 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci a30a4df9c0 Cleanup config and add tests 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 1e0d5aa91d fs_wait_for_mount cleanup 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci d09fadbc28 fs_inode cleanup 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 21ce889449 Fixup listxattr error handling 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 9d1a4696e9 Small improvements to fs::statvfs_cache 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci ed0cbcf064 Cleanup fuse_readlink 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 076e359891 Improve thread pool 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci bf98d3f3a6 Fix possible error handling issue with errno 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci c45c5ce920 Make cfgfile depth calculation thread friendly 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci e0a892f0d0 Improve error handling when querying pagesize 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 164b7f73c9 Misc fixes and improvements to thread pool 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 3a45be4cd8 Rework string utils, more test cases 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 107a3acf95 Better error handling in pagesize config type 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci a5eafb8972 Check string is not empty before resize in Config::keys 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 8199cc8a8b Fix error handling in fuse chmod, chown, setxattr, removexattr 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci e5d897f3dc Ensure zero initialized timeouts structs 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci c0af09f7d8 Fix incorrect type in fs_read and write 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 1b1e14ca29 Remove dead branch in fuse_reply_data 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 82b89aa099 Change session->exited to sig_atomic_t 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci aa91e1b2ba Initialize stat buffer and add stat call in fuse_mount_sys 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 82e81afcdc Add NULL check for cmsg in receive_fd to prevent segfault 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 1a5c39f813 Fix buffer overflow in readlink by reserving space for null terminator 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci fce1803d42 Use RND class for shrink_to_rand_elem and add assertions 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 5a9a17afcb fix objpool: add size tracking, improve allocation matching; fix msgbuf: clear pool on size change 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 8648de6bb6 fix policy_eppfrd.cpp: return 0 when branchinfo found, -ENOENT only when empty 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci f227a63e15 fix str.cpp: use string_view::npos, fix erase() calls, remove extra whitespace 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci aa47b74dea Remove iterator methods from Branches class 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 206ad1dd53 Fix futimens bugs: stat pointer, variable names, and negation 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 80aead7085 Add null pointer checks for FileInfo/DirInfo 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 1220befacf Fix copydata_copy_file_range bug 2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 01aa888c7a Fix eplus search bug
Not that anyone would ever use this but it was noticed in a scan.
2026-03-24 21:13:34 -04:00
Antonio SJ Musumeci 6a6aa054cd Add support for Arch, btw 2026-03-14 11:39:48 -04:00
Antonio SJ Musumeci 04939bbe72 No longer explicitly error out on broken mount point
If ENOTCONN (broken FUSE mount) umount first and continue to mount.
2026-03-08 00:39:45 -05:00
Antonio SJ Musumeci 2ea6290e4b Update limiting drive spinup doc 2026-03-04 11:05:55 -05:00
Antonio SJ Musumeci c66c68b137 Misc cleanup 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 075a8a6548 Convert some push_back to emplace_back 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 3c6d1b47f6 Remove unused "using"s 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 6f281b9f7f Remove duplicate includes 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 511bf47af9 Convert stdint to short ints 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 53c5a2cf1e Move fmtlib to vendored and upgrade to 12.1.0 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci eb8074ed9a Update vendored/nonstd to latest and fix API usage 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 38c19df95a Update vendored/boost to 1.90.0 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci b1228f11bd Update vendored/acutest to latest master 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci cfafafd7ff Update vendored/CLI11 to v2.6.2 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci c8ce82ce54 Update vendored/subprocess to v2.5 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 05c3d97368 Update vendored/rapidhash to v3 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 2c87e68635 Cleanup ugid.hpp usage 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 8f1ef59770 Move 3rd party libraries to "vendored" directory 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci a460c52861 Remove unused files 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci a4aade6666 Make global inode calculation function pointer atomic 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 1d1bcd3b87 Add extra checks in for file handle lookup 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 099ce2d91d Add more asserts 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 56cd831ec8 Fix str::trim to handle all whitespace characters
Use std::isspace() instead of comparing only to space character.
Now handles tabs, newlines, carriage returns, etc.
2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci e43be3c221 Remove unused PolicyCache implementation 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 2754536727 Fix integer overflow in size suffix parsing
Use __builtin_mul_overflow to detect overflow when applying size
suffixes (K, M, G, T) in str::from for int64_t and uint64_t. Previously,
large values with suffixes could silently wrap around.
2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci a0cff637e1 Misc cleanup 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci a5a6c892b9 Update copyright headers 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 3d50d4a52f Explicitly return EXIT_FAILURE 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci f69c31fd77 Rework objpool 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci a2860c3f1f Use debug mutex more 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 3f70e66ee3 Rework node obj pool to use objpool 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 160d3c832d Fix branches unit test 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 15bcc47d6a Remove "channels" and embed into session 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci e55bb93890 Simplify lowlevel fuse state management and req mem 2026-03-02 01:09:16 -05:00
Antonio SJ Musumeci 5a2a9e4530 Remove fuse_ll_pipe dead code 2026-03-02 01:09:16 -05:00
trapexit b4239495a0 Update docs for logging changes + new video (#1618) 2026-02-12 23:21:43 -06:00
trapexit 3222de6c9c Fixup debug output (#1617) 2026-02-12 22:50:38 -06:00
Aaron Andersen be4d388813 Add support for -n to not update mtab (#1616) 2026-02-12 20:35:48 -06:00
trapexit a47842e2f8 Add MOS to projects and update misc links (#1614) 2026-01-27 11:25:32 -06:00
trapexit f769d7474b Add new links to media docs (#1613) 2026-01-27 07:59:38 -06:00
trapexit 2ae57642f4 Misc documentation updates (#1612) 2026-01-27 07:00:22 -06:00
trapexit 7f1b7ac343 Add lshw to collect info tool (#1610) 2026-01-19 13:57:42 -06:00
996 changed files with 28936 additions and 17433 deletions
-40
View File
@@ -1,40 +0,0 @@
name: "mkdocs"
on:
push:
branches:
- master
tags:
- '*'
permissions:
contents: write
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.x'
- name: Configure Git Credentials
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
- name: Install dependencies
working-directory: ./mkdocs
run: |
./build-venv
- name: Deploy docs
working-directory: ./mkdocs
env:
REF_TYPE: ${{ github.ref_type }}
REF_NAME: ${{ github.ref_name }}
run: |
./generate-and-push-docs
+1
View File
@@ -9,6 +9,7 @@
* scope_guard: https://github.com/Neargye/scope_guard
* subprocess: https://github.com/arun11299/cpp-subprocess
* CLI11: https://github.com/CLIUtils/CLI11
* string-lite: https://github.com/martinmoene/string-lite
* boost
* download boost
* ./bootstrap.sh
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2025, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+58 -34
View File
@@ -29,6 +29,8 @@ MKDIR ?= mkdir
MKTEMP ?= mktemp
MV ?= mv
RM ?= rm
RPMBUILD ?= rpmbuild
RPMBUILD_FLAGS ?=
SED ?= sed
STRIP ?= strip
TAR ?= tar
@@ -36,7 +38,8 @@ TOUCH ?= touch
BUILDDIR := build
GITHUB_REPO ?= "https://github.com/trapexit/mergerfs"
RELEASE_SAMPLE ?= "debian:13.amd64"
DEFAULT_TARGET := debian:13.amd64
ifndef GIT_REPO
ifneq ($(shell $(GIT) --version 2> /dev/null),)
@@ -47,26 +50,30 @@ endif
endif
USE_XATTR ?= 1
UGID_USE_RWLOCK ?= 0
ifdef NDEBUG
OPT_FLAGS := -O2 -DNDEBUG
else
ifdef SANITIZE
ifeq ($(SANITIZE),1)
override SANITIZE := -fsanitize=address,undefined,leak
else
else ifeq ($(filter -fsanitize=%,$(SANITIZE)),)
override SANITIZE := -fsanitize=$(SANITIZE)
endif
endif
OPT_FLAGS := -O0 \
-g \
-fno-omit-frame-pointer \
$(SANITIZE) \
-fstack-protector-strong \
-Wextra \
-Wno-unused-parameter \
-DDEBUG
COMMON_OPT_FLAGS += \
-fstack-protector-strong \
-Wextra \
-Wno-unused-parameter
DEBUG_OPT_FLAGS += \
-O0 \
-g3 \
-fno-omit-frame-pointer \
$(SANITIZE)
ifdef RELEASE
OPT_FLAGS := -O3 $(COMMON_OPT_FLAGS)
else
OPT_FLAGS ?= $(DEBUG_OPT_FLAGS) $(COMMON_OPT_FLAGS)
endif
ifdef STATIC
@@ -94,7 +101,8 @@ TESTS_DEPS += $(DEPS)
MANPAGE := mergerfs.1
CPPFLAGS ?=
override CPPFLAGS += \
-D_FILE_OFFSET_BITS=64
-D_FILE_OFFSET_BITS=64 \
-DBOOST_UNORDERED_DISABLE_PARALLEL_ALGORITHMS
CFLAGS ?= \
$(OPT_FLAGS) \
-Wall \
@@ -115,20 +123,22 @@ CXXFLAGS ?= \
-Wno-unused-result \
-pipe
override CXXFLAGS += \
-std=c++17 \
-std=c++20 \
-MMD \
-MP
override INC_FLAGS := \
-Isrc \
-Ilibfuse/include
-Ivendored \
-Ivendored/libfuse/include
override MFS_FLAGS := \
-DUSE_XATTR=$(USE_XATTR) \
-DUGID_USE_RWLOCK=$(UGID_USE_RWLOCK)
-DUSE_XATTR=$(USE_XATTR)
override TESTS_FLAGS := \
-Isrc \
-Ivendored \
-Ivendored/acutest \
-DTESTS
LIBFUSE := libfuse/$(BUILDDIR)/libfuse.a
LIBFUSE := vendored/libfuse/$(BUILDDIR)/libfuse.a
LDFLAGS ?=
LDLIBS := \
-lrt \
@@ -169,6 +179,8 @@ help:
@echo "USE_XATTR=0 - build program without xattrs functionality"
@echo "STATIC=1 - build static binary"
@echo "LTO=1 - build with link time optimization"
@echo "SANITIZE=1 - build with sanitizers (address,undefined,leak)"
@echo "SANITIZE=list - build with custom sanitizers (comma separated)"
@echo "CLEANUP=1 - cleanup images between release build process"
@echo "PKGDIR=/dir/ - location for release build pkgs"
@echo "GITREF=gitref - gitref to use for release builds"
@@ -177,19 +189,21 @@ help:
$(BUILDDIR)/mergerfs: $(LIBFUSE) src/version.hpp $(OBJS)
$(CXX) $(CXXFLAGS) $(INC_FLAGS) $(MFS_FLAGS) $(CPPFLAGS) $(OBJS) -o $@ $(LDFLAGS) $(LDLIBS)
$(BUILDDIR)/fsck.mergerfs:
$(BUILDDIR)/fsck.mergerfs: $(BUILDDIR)/mergerfs
$(LN) -sf "mergerfs" $@
$(BUILDDIR)/mergerfs.collect-info:
$(BUILDDIR)/mergerfs.collect-info: $(BUILDDIR)/mergerfs
$(LN) -sf "mergerfs" $@
$(BUILDDIR)/tests: $(BUILDDIR)/mergerfs $(TESTS_OBJS)
$(CXX) $(CXXFLAGS) $(TESTS_FLAGS) $(INC_FLAGS) $(MFS_FLAGS) $(CPPFLAGS) $(TESTS_OBJS) -o $@ $(LDFLAGS) $(LDLIBS)
.PHONY: libfuse
libfuse: $(LIBFUSE)
$(MAKE) -C vendored/libfuse
$(LIBFUSE):
libfuse:
$(MAKE) -C libfuse
$(MAKE) -C vendored/libfuse $(BUILDDIR)/libfuse.a RELEASE=$(RELEASE)
tests: $(BUILDDIR)/tests
@@ -215,7 +229,7 @@ $(BUILDDIR)/stamp:
$(BUILDDIR)/.objs/%.cpp.o: src/%.cpp $(BUILDDIR)/stamp
$(CXX) $(CXXFLAGS) $(INC_FLAGS) $(MFS_FLAGS) $(CPPFLAGS) -c $< -o $@
$(BUILDDIR)/.test_objs/%.cpp.o: tests/%.cpp
$(BUILDDIR)/.test_objs/%.cpp.o: tests/%.cpp | $(BUILDDIR)/stamp
$(CXX) $(CXXFLAGS) $(TESTS_FLAGS) $(INC_FLAGS) $(MFS_FLAGS) $(CPPFLAGS) -c $< -o $@
$(BUILDDIR)/preload.so: $(BUILDDIR)/stamp tools/preload.c
@@ -227,7 +241,7 @@ preload: $(BUILDDIR)/preload.so
clean: rpm-clean
$(RM) -rf $(BUILDDIR)
$(FIND) . -name "*~" -delete
$(MAKE) -C libfuse clean
$(MAKE) -C vendored/libfuse clean
.PHONY: distclean
distclean: clean
@@ -248,7 +262,7 @@ install-base: all
.PHONY: install-mount-tools
install-mount-tools: install-base
$(MKDIR) -p "$(INSTALLBINDIR)"
$(MAKE) -C libfuse install
$(MAKE) -C vendored/libfuse install
.PHONY: install-man
install-man: man/$(MANPAGE)
@@ -265,10 +279,12 @@ install-strip: install-base
$(STRIP) "$(INSTALLBINDIR)/mergerfs"
.PHONY: uninstall
uninstall: uninstall-base uninstall-mount.mergerfs uninstall-man
uninstall: uninstall-base uninstall-mount.mergerfs uninstall-preload uninstall-man
uninstall-base:
$(RM) -f "$(INSTALLBINDIR)/mergerfs"
$(RM) -f "$(INSTALLBINDIR)/fsck.mergerfs"
$(RM) -f "$(INSTALLBINDIR)/mergerfs.collect-info"
uninstall-mount.mergerfs:
$(RM) -f "$(INSTALLBINDIR)/mount.mergerfs"
@@ -276,6 +292,9 @@ uninstall-mount.mergerfs:
uninstall-man:
$(RM) -f "$(INSTALLMAN1DIR)/$(MANPAGE)"
uninstall-preload:
$(RM) -f "$(INSTALLLIBDIR)/preload.so"
.PHONY: tarball
tarball: changelog version
$(eval VERSION := $(shell cat VERSION))
@@ -326,7 +345,7 @@ rpm: tarball
$(SED) 's/__VERSION__/$(VERSION)/g' mergerfs.spec > \
rpmbuild/SOURCES/mergerfs.spec
cp -ar mergerfs-$(VERSION).tar.gz rpmbuild/SOURCES
rpmbuild -ba rpmbuild/SOURCES/mergerfs.spec \
$(RPMBUILD) $(RPMBUILD_FLAGS) -ba rpmbuild/SOURCES/mergerfs.spec \
--define "_topdir $(CURDIR)/rpmbuild"
.PHONY: install-build-pkgs
@@ -335,7 +354,7 @@ install-build-pkgs:
.PHONY: install-build-tools
install-build-tools:
./bulidtools/install-build-tools
./buildtools/install-build-tools
define build_release
$(eval GITREF ?= $(shell git describe --exact-match --tags HEAD 2>/dev/null || git symbolic-ref --short HEAD 2>/dev/null || git rev-parse --short HEAD))
@@ -348,13 +367,16 @@ define build_release
--branch=$(GITREF)
endef
.PHONY: release
.PHONY: release release-all
.PHONY: release-amd64 release-arm64 release-armhf release-riscv64
.PHONY: release-sample release-static release-tarball
.PHONY: release-sample release-static release-tarball release-installer
release:
$(MAKE) RELEASE=1 CPPFLAGS="$(CPPFLAGS) -DNDEBUG" all
release-all:
$(call build_release,"all")
release-sample:
$(call build_release,$(RELEASE_SAMPLE))
$(if $(TARGET),,$(eval TARGET := $(shell ./buildtools/detect-local-target 2>/dev/null)))
$(call build_release,$(if $(TARGET),$(TARGET),$(DEFAULT_TARGET)))
release-amd64:
$(call build_release,"amd64")
release-arm64:
@@ -362,11 +384,13 @@ release-arm64:
release-armhf:
$(call build_release,"armhf")
release-riscv64:
$(call build_release,"risc64")
$(call build_release,"riscv64")
release-static:
$(call build_release,"static")
release-tarball:
$(call build_release,"tarball")
release-installer:
$(call build_release,"installer")
container:
$(eval GITREF ?= $(shell git describe --exact-match --tags HEAD 2>/dev/null || git symbolic-ref --short HEAD 2>/dev/null || git rev-parse --short HEAD))
+31 -10
View File
@@ -1,6 +1,19 @@
#!/bin/sh
set -e
if [ -e /tmp/build-env ]; then
. /tmp/build-env
fi
rpmbuild_flags()
{
case "$(command -v "${CXX:-g++}")" in
/opt/rh/*)
printf '%s' '--undefine _annotated_build'
;;
esac
}
if [ $# -ne 2 ]; then
echo "usage: $0 <branch> <git_repo>"
echo "argv: $@"
@@ -28,28 +41,36 @@ mkdir "/build"
if [ -e /usr/bin/apt-get ]; then
make deb
elif [ -e /usr/bin/dnf ]; then
make rpm
make RPMBUILD_FLAGS="$(rpmbuild_flags)" rpm
elif [ -e /usr/bin/yum ]; then
. /opt/rh/devtoolset-9/enable
make rpm
make RPMBUILD_FLAGS="$(rpmbuild_flags)" rpm
elif [ -e /sbin/apk ]; then
echo "NOT YET SUPPORTED"
exit 1
elif [ -e /usr/sbin/pkg ]; then
echo "NOT YET SUPPORTED"
exit 1
elif [ -e /usr/bin/pacman ]; then
make tarball
else
echo "NOT YET SUPPORTED"
exit 1
fi
echo "Copy packages to host..."
find "${BUILDDIR}" \
-type f \
\( -name "*.deb" -or -name "*.rpm" \) \
-not -name "*dbgsym*" \
-not -name "*sym*" \
-not -name "*src.rpm" \
-exec cp -v {} /build/ \;
if [ -e /usr/bin/pacman ]; then
find "${BUILDDIR}" \
-type f \
-name "*.tar.gz" \
-exec cp -v {} /build/ \;
else
find "${BUILDDIR}" \
-type f \
\( -name "*.deb" -or -name "*.rpm" \) \
-not -name "*dbgsym*" \
-not -name "*sym*" \
-not -name "*src.rpm" \
-exec cp -v {} /build/ \;
fi
exit 0
+49 -10
View File
@@ -18,10 +18,12 @@ def build(git_repo,
'--force-rm']
if os.path.exists(git_repo):
git_repo = os.path.realpath(git_repo)
args += ['-v',f'{git_repo}:/tmp/mergerfs.git:ro']
args += ['-v',f'{git_repo}:/tmp/mergerfs.git:ro:z']
args += ['--build-arg=GIT_REPO=file:///tmp/mergerfs.git']
else:
args += [f'--build-arg=GIT_REPO={git_repo}']
os.makedirs(pkgdirpath,exist_ok=True)
args += ['-v',f'{os.path.realpath(pkgdirpath)}:/pkgs:ro']
args += ['-o',pkgdirpath,
'-f',containerfile,
f'--build-arg=BRANCH={branch}',
@@ -30,8 +32,6 @@ def build(git_repo,
# TODO: Capture output and write to log
print(args)
rv = subprocess.run(args)
os.makedirs(pkgdirpath,exist_ok=True)
report_filepath = os.path.join(pkgdirpath,"build-report.txt")
with open(report_filepath,"a+") as f:
build = os.path.basename(containerfile)
@@ -47,14 +47,49 @@ def build(git_repo,
def setup():
args = ['sudo',
'apt-get',
'install',
'-fy',
'qemu-user-static',
'qemu-user-binfmt']
# Detect distribution
if os.path.exists('/usr/bin/apt-get'):
args = ['sudo',
'apt-get',
'install',
'-fy',
'qemu-user-static',
'qemu-user-binfmt']
elif os.path.exists('/usr/bin/pacman'):
args = ['sudo',
'pacman',
'-Sy',
'--noconfirm',
'qemu-user-static']
else:
print("Unsupported distribution for setup")
sys.exit(1)
print(args)
subprocess.run(args)
setup_binfmt()
def setup_binfmt():
"""Register qemu-user-static binfmt handlers (needed on Arch, etc.)."""
binfmt_conf = """\
:qemu-aarch64:M::\\x7fELF\\x02\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\xb7\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff\\xff:/usr/bin/qemu-aarch64-static:F
:qemu-arm:M::\\x7fELF\\x01\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x28\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff\\xff:/usr/bin/qemu-arm-static:F
:qemu-armeb:M::\\x7fELF\\x01\\x02\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x28:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff:/usr/bin/qemu-armeb-static:F
:qemu-riscv64:M::\\x7fELF\\x02\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\xf3\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff\\xff:/usr/bin/qemu-riscv64-static:F
:qemu-i386:M::\\x7fELF\\x01\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x03\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff\\xff:/usr/bin/qemu-i386-static:F
"""
conf_path = '/etc/binfmt.d/qemu-static.conf'
print(f"Writing {conf_path}")
try:
with open(conf_path, 'w') as f:
f.write(binfmt_conf)
except PermissionError:
print(f"Permission denied writing {conf_path}; trying with sudo...")
subprocess.run(['sudo', 'tee', conf_path],
input=binfmt_conf.encode(), check=True)
print("Restarting systemd-binfmt...")
subprocess.run(['sudo', 'systemctl', 'restart', 'systemd-binfmt'])
def podman_cleanup():
@@ -81,7 +116,7 @@ def parse_args():
required=False,
action='store_true')
p.add_argument('--git-repo',
required=True,
required=False,
type=str)
return p.parse_args()
@@ -101,6 +136,10 @@ def main():
setup()
sys.exit(0)
if not args.git_repo:
print("error: the following arguments are required: --git-repo")
sys.exit(1)
containerfiles = []
basepath = 'buildtools/containerfiles'
if os.path.exists(f'{basepath}/{args.target}'):
+10
View File
@@ -0,0 +1,10 @@
FROM --platform=amd64 fedora:44 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
+10
View File
@@ -0,0 +1,10 @@
FROM --platform=arm64 fedora:44 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
+7 -3
View File
@@ -1,9 +1,13 @@
ARG BUILD_TIMESTAMP=0
FROM --platform=linux/amd64 ubuntu:latest as build
RUN apt-get update && apt-get install -y makeself
COPY install-build-pkgs /tmp/
RUN /tmp/install-build-pkgs && \
DEBIAN_FRONTEND=noninteractive apt-get -qy --no-install-suggests --no-install-recommends install \
makeself
RUN mkdir -p /tmp/mergerfs/
RUN --mount=type=bind,src=pkgs,dst=/pkgs ls -lhd /pkgs
RUN --mount=type=bind,src=pkgs,dst=/pkgs cp -v /pkg/*static*.gz /tmp/mergerfs/
RUN ls -lhd /pkgs
RUN cp -v /pkgs/*static*.gz /tmp/mergerfs/
RUN mkdir -p /build
RUN makeself /tmp/mergerfs/ /build/mergerfs-installer.run "mergerfs" echo "foo"
FROM scratch
@@ -0,0 +1,10 @@
FROM --platform=amd64 ubuntu:26.04 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
@@ -0,0 +1,10 @@
FROM --platform=arm64 ubuntu:26.04 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
@@ -0,0 +1,10 @@
FROM --platform=armhf ubuntu:26.04 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
@@ -0,0 +1,10 @@
FROM --platform=riscv64 ubuntu:26.04 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
+119
View File
@@ -0,0 +1,119 @@
#!/bin/bash
# Detects local OS and maps to containerfiles target
# Returns target if found in buildtools/containerfiles/, empty string otherwise
set -euo pipefail
CONTAINERFILES_DIR="buildtools/containerfiles"
# Detect system architecture and map to containerfiles format
detect_arch() {
local arch
arch=$(uname -m)
case "$arch" in
x86_64) echo "amd64" ;;
aarch64) echo "arm64" ;;
armv7l) echo "armhf" ;;
i686|i386) echo "i386" ;;
riscv64) echo "riscv64" ;;
*) echo "" ;;
esac
}
# Detect distro and version from os-release
detect_distro() {
local os_release="${1:-}"
if [[ ! -r "$os_release" ]]; then
return 1
fi
# Source the file safely - ID and VERSION_ID are environment variables
. "$os_release"
local distro="$ID"
local version="$VERSION_ID"
# Map distro ID to containerfiles naming convention
case "$distro" in
ubuntu|debian|fedora) ;;
rocky|rhel|centos|almalinux) distro="rockylinux" ;;
*)
# Fallback to ID_LIKE for derivatives
if [[ -n "${ID_LIKE:-}" ]]; then
case "$ID_LIKE" in
*ubuntu*) distro="ubuntu" ;;
*debian*) distro="debian" ;;
*fedora*) distro="fedora" ;;
esac
fi
;;
esac
# Only return if we have valid distro and version
if [[ -n "$distro" && -n "$version" ]]; then
echo "$distro:$version"
else
return 1
fi
}
# Try to find a matching containerfile target
find_containerfile() {
local target="$1"
local containerfile="${CONTAINERFILES_DIR}/${target}"
# Direct match
if [[ -d "$containerfile" || -f "$containerfile" ]]; then
echo "$target"
return 0
fi
return 1
}
# Main detection logic
main() {
local arch
# Detect architecture
arch=$(detect_arch)
if [[ -z "$arch" ]]; then
echo "WARNING: Could not detect system architecture" >&2
exit 1
fi
# Try to detect distro from multiple locations
local distro_ver
distro_ver=$(detect_distro "/etc/os-release") || \
distro_ver=$(detect_distro "/usr/lib/os-release") || {
echo "WARNING: Could not detect distro from os-release" >&2
exit 1
}
# Build target string
local target="${distro_ver}.${arch}"
# Try to find matching containerfile
if find_containerfile "$target"; then
exit 0
fi
# Try with partial version matching (e.g., 25.10 → 25)
local distro="${distro_ver%%:*}"
local version="${distro_ver##*:}"
# Strip last minor version number and try again
local short_version="${version%.*}"
if [[ -n "$short_version" ]]; then
target="${distro}:${short_version}.${arch}"
if find_containerfile "$target"; then
exit 0
fi
fi
# No match found
echo "WARNING: No containerfile found for target: ${distro_ver}.${arch}" >&2
exit 1
}
main "$@"
+128 -3
View File
@@ -1,37 +1,162 @@
#!/bin/sh
set -e
BUILD_ENV=/tmp/build-env
write_gcc_env()
{
{
printf 'export CC=%s\n' "$1"
printf 'export CXX=%s\n' "$2"
} > "$BUILD_ENV"
}
write_toolset_env()
{
{
printf '. %s\n' "$1"
printf 'export CC=gcc\n'
printf 'export CXX=g++\n'
} > "$BUILD_ENV"
}
latest_apt_gxx()
{
apt-cache --names-only search '^g[+][+]-[0-9]' \
| sed -n 's/^\(g++-[0-9][0-9]*\) .*/\1/p' \
| sort -t- -k2,2V \
| tail -n 1
}
latest_dnf_toolset()
{
dnf -q list --showduplicates 'gcc-toolset-*-gcc-c++' 2>/dev/null \
| sed -n 's/^gcc-toolset-\([0-9][0-9]*\)-gcc-c++\..*/\1/p' \
| sort -V \
| tail -n 1
}
latest_yum_toolset()
{
yum -q list available 'devtoolset-*-gcc-c++' 2>/dev/null \
| sed -n 's/^devtoolset-\([0-9][0-9]*\)-gcc-c++\..*/\1/p' \
| sort -V \
| tail -n 1
}
version_gt()
{
highest="$(printf '%s\n%s\n' "$1" "$2" | sort -V | tail -n 1)"
[ "$highest" = "$1" ] && [ "$1" != "$2" ]
}
check_cxx20()
{
cxx="${CXX:-g++}"
test_src=/tmp/mergerfs-cxx20-check.cpp
test_bin=/tmp/mergerfs-cxx20-check
cat > "$test_src" <<'EOF'
template <typename T>
concept Incrementable = requires(T value) { value + 1; };
static_assert(Incrementable<int>);
int main()
{
return 0;
}
EOF
if ! "$cxx" -std=c++20 "$test_src" -o "$test_bin" >/dev/null 2>&1; then
echo "error: $cxx does not support C++20" >&2
exit 1
fi
rm -f "$test_src" "$test_bin"
}
: > "$BUILD_ENV"
if [ -e /usr/bin/apt-get ]; then
export DEBIAN_FRONTEND=noninteractive
apt-get -qy update
cxx_pkg="$(latest_apt_gxx)"
if [ -n "$cxx_pkg" ]; then
gcc_pkg="gcc-${cxx_pkg#g++-}"
compiler_pkgs="$gcc_pkg $cxx_pkg"
write_gcc_env "$gcc_pkg" "$cxx_pkg"
else
compiler_pkgs="g++"
write_gcc_env gcc g++
fi
apt-get -qy --force-yes --no-install-suggests --no-install-recommends \
install \
ca-certificates \
build-essential \
dpkg-dev \
git \
g++ \
$compiler_pkgs \
debhelper \
lsb-release \
fakeroot
. "$BUILD_ENV"
check_cxx20
elif [ -e /usr/bin/dnf ]; then
dnf -y update
toolset="$(latest_dnf_toolset)"
dnf -y install \
git rpm-build gcc-c++ make which libatomic
default_gxx_major="$(g++ -dumpversion | cut -f1 -d.)"
if [ -n "$toolset" ] && version_gt "$toolset" "$default_gxx_major"; then
dnf -y install \
"gcc-toolset-${toolset}-gcc" \
"gcc-toolset-${toolset}-gcc-c++"
write_toolset_env "/opt/rh/gcc-toolset-${toolset}/enable"
else
write_gcc_env gcc g++
fi
. "$BUILD_ENV"
check_cxx20
elif [ -e /usr/bin/yum ]; then
yum -y update
yum -y install \
git rpm-build gcc-c++ make which
yum -y install centos-release-scl
yum -y install devtoolset-9-gcc\*
yum -y install centos-release-scl || true
yum -y update
toolset="$(latest_yum_toolset)"
default_gxx_major="$(g++ -dumpversion | cut -f1 -d.)"
if [ -n "$toolset" ] && version_gt "$toolset" "$default_gxx_major"; then
yum -y install \
"devtoolset-${toolset}-gcc" \
"devtoolset-${toolset}-gcc-c++"
write_toolset_env "/opt/rh/devtoolset-${toolset}/enable"
else
write_gcc_env gcc g++
fi
. "$BUILD_ENV"
check_cxx20
elif [ -e /usr/bin/zypper ]; then
zypper update -y
zypper install -y \
git rpm-build gcc-c++ make which
write_gcc_env gcc g++
. "$BUILD_ENV"
check_cxx20
elif [ -e /sbin/apk ]; then
apk add \
abuild git gcc g++ make \
linux-headers
write_gcc_env gcc g++
. "$BUILD_ENV"
check_cxx20
elif [ -e /usr/sbin/pkg ]; then
pkg install \
git gmake gcc
elif [ -e /usr/bin/pacman ]; then
pacman -Sy --noconfirm base-devel git fakeroot
write_gcc_env gcc g++
. "$BUILD_ENV"
check_cxx20
fi
+18 -5
View File
@@ -1,19 +1,32 @@
#!/bin/sh
if [ "$(id -u)" -eq 0 ]; then
SUDO=
elif command -v sudo >/dev/null 2>&1; then
SUDO=sudo
elif command -v doas >/dev/null 2>&1; then
SUDO=doas
else
printf '%s\n' "error: need root privileges; run as root or install sudo/doas" >&2
exit 1
fi
if [ -e /usr/bin/apt-get ]; then
export DEBIAN_FRONTEND=noninteractive
apt-get -qy update
apt-get -qy --force-yes --no-install-suggests --no-install-recommends install \
${SUDO} apt-get -qy update
${SUDO} apt-get -qy --no-install-suggests --no-install-recommends install \
qemu-user-static \
podman
elif [ -e /usr/bin/dnf ]; then
dnf -y update
dnf -y install \
${SUDO} dnf -y update
${SUDO} dnf -y install \
qemu-user-static \
podman
elif [ -e /sbin/apk ]; then
apk add \
${SUDO} apk add \
python3 \
qemu-user-static \
podman
elif [ -e /usr/bin/pacman ]; then
${SUDO} pacman -Sy --noconfirm qemu-user-static podman
fi
+4 -1
View File
@@ -6,8 +6,11 @@ export DH_VERBOSE=1
%:
dh $@
override_dh_auto_build:
$(MAKE) release
override_dh_auto_install:
$(MAKE) \
DESTDIR=$(CURDIR)/debian/mergerfs \
PREFIX=/usr \
install
install
-15
View File
@@ -1,15 +0,0 @@
#pragma once
#include "fuse_req_ctx.h"
struct fuse_ll;
struct fuse_chan;
typedef struct fuse_req_t fuse_req_t;
struct fuse_req_t
{
fuse_req_ctx_t ctx;
struct fuse_ll *f;
struct fuse_chan *ch;
unsigned int ioctl_64bit : 1;
};
-7
View File
@@ -1,7 +0,0 @@
#pragma once
#ifdef DEBUG
#include "mutex_debug.hpp"
#else
#include "mutex_ndebug.hpp"
#endif
-389
View File
@@ -1,389 +0,0 @@
#pragma once
#include "moodycamel/blockingconcurrentqueue.h"
#include "moodycamel/lightweightsemaphore.h"
#include "invocable.h"
#include <algorithm>
#include <atomic>
#include <csignal>
#include <cstring>
#include <future>
#include <memory>
#include <mutex>
#include <stdexcept>
#include <string>
#include <thread>
#include <vector>
#include <utility>
#include <pthread.h>
#include <syslog.h>
#define SEM
//#undef SEM
#ifdef SEM
#define SEMA_WAIT(S) (S.wait())
#define SEMA_SIGNAL(S) (S.signal())
#else
#define SEMA_WAIT(S)
#define SEMA_SIGNAL(S)
#endif
struct ThreadPoolTraits : public moodycamel::ConcurrentQueueDefaultTraits
{
static const int MAX_SEMA_SPINS = 1;
};
class ThreadPool
{
public:
using PToken = moodycamel::ProducerToken;
using CToken = moodycamel::ConsumerToken;
private:
using Func = ofats::any_invocable<void()>;
using Queue = moodycamel::BlockingConcurrentQueue<Func,ThreadPoolTraits>;
public:
explicit
ThreadPool(const unsigned thread_count_ = std::thread::hardware_concurrency(),
const unsigned max_queue_depth_ = std::thread::hardware_concurrency(),
std::string const name_ = {});
~ThreadPool();
private:
static void *start_routine(void *arg_);
public:
int add_thread(std::string const name = {});
int remove_thread(void);
int set_threads(const std::size_t count);
void shutdown(void);
public:
template<typename FuncType>
void
enqueue_work(ThreadPool::PToken &ptok_,
FuncType &&func_);
template<typename FuncType>
void
enqueue_work(FuncType &&func_);
template<typename FuncType>
[[nodiscard]]
std::future<std::invoke_result_t<FuncType>>
enqueue_task(FuncType&& func_);
public:
std::vector<pthread_t> threads() const;
ThreadPool::PToken ptoken();
private:
Queue _queue;
moodycamel::LightweightSemaphore _sema;
private:
std::string const _name;
std::vector<pthread_t> _threads;
mutable std::mutex _threads_mutex;
};
inline
ThreadPool::ThreadPool(const unsigned thread_count_,
const unsigned max_queue_depth_,
const std::string name_)
: _queue(),
_sema(max_queue_depth_),
_name(name_)
{
sigset_t oldset;
sigset_t newset;
sigfillset(&newset);
pthread_sigmask(SIG_BLOCK,&newset,&oldset);
_threads.reserve(thread_count_);
for(std::size_t i = 0; i < thread_count_; ++i)
{
int rv;
pthread_t t;
rv = pthread_create(&t,NULL,ThreadPool::start_routine,this);
if(rv != 0)
{
syslog(LOG_WARNING,
"threadpool (%s): error spawning thread - %d (%s)",
_name.c_str(),
rv,
strerror(rv));
continue;
}
if(!_name.empty())
pthread_setname_np(t,_name.c_str());
_threads.push_back(t);
}
pthread_sigmask(SIG_SETMASK,&oldset,NULL);
if(_threads.empty())
throw std::runtime_error("threadpool: failed to spawn any threads");
syslog(LOG_DEBUG,
"threadpool (%s): spawned %zu threads w/ max queue depth %u",
_name.c_str(),
_threads.size(),
max_queue_depth_);
}
inline
ThreadPool::~ThreadPool()
{
syslog(LOG_DEBUG,
"threadpool (%s): destroying %zu threads",
_name.c_str(),
_threads.size());
for(auto t : _threads)
pthread_cancel(t);
for(auto t : _threads)
pthread_join(t,NULL);
}
inline
void*
ThreadPool::start_routine(void *arg_)
{
ThreadPool *btp = static_cast<ThreadPool*>(arg_);
bool done;
ThreadPool::Func func;
ThreadPool::Queue &q = btp->_queue;
moodycamel::LightweightSemaphore &sema = btp->_sema;
ThreadPool::CToken ctok(btp->_queue);
done = false;
while(!done)
{
q.wait_dequeue(ctok,func);
pthread_setcancelstate(PTHREAD_CANCEL_DISABLE,NULL);
try
{
func();
}
catch(std::exception &e)
{
done = true;
}
SEMA_SIGNAL(sema);
pthread_setcancelstate(PTHREAD_CANCEL_ENABLE,NULL);
}
return nullptr;
}
inline
int
ThreadPool::add_thread(const std::string name_)
{
int rv;
pthread_t t;
sigset_t oldset;
sigset_t newset;
std::string name;
name = (name_.empty() ? _name : name_);
sigfillset(&newset);
pthread_sigmask(SIG_BLOCK,&newset,&oldset);
rv = pthread_create(&t,NULL,ThreadPool::start_routine,this);
pthread_sigmask(SIG_SETMASK,&oldset,NULL);
if(rv != 0)
{
syslog(LOG_WARNING,
"threadpool (%s): error spawning thread - %d (%s)",
_name.c_str(),
rv,
strerror(rv));
return -rv;
}
if(!name.empty())
pthread_setname_np(t,name.c_str());
{
std::lock_guard<std::mutex> lg(_threads_mutex);
_threads.push_back(t);
}
return 0;
}
inline
int
ThreadPool::remove_thread(void)
{
{
std::lock_guard<std::mutex> lg(_threads_mutex);
if(_threads.size() <= 1)
return -EINVAL;
}
std::promise<pthread_t> promise;
auto func =
[this,&promise]()
{
pthread_t t;
t = pthread_self();
promise.set_value(t);
{
std::lock_guard<std::mutex> lg(_threads_mutex);
for(auto i = _threads.begin(); i != _threads.end(); ++i)
{
if(*i != t)
continue;
_threads.erase(i);
break;
}
}
pthread_exit(NULL);
};
enqueue_work(std::move(func));
pthread_join(promise.get_future().get(),NULL);
return 0;
}
inline
int
ThreadPool::set_threads(std::size_t const count_)
{
int diff;
{
std::lock_guard<std::mutex> lg(_threads_mutex);
diff = ((int)count_ - (int)_threads.size());
}
for(auto i = diff; i > 0; --i)
add_thread();
for(auto i = diff; i < 0; ++i)
remove_thread();
return diff;
}
inline
void
ThreadPool::shutdown(void)
{
std::lock_guard<std::mutex> lg(_threads_mutex);
for(pthread_t tid : _threads)
pthread_cancel(tid);
for(pthread_t tid : _threads)
pthread_join(tid,NULL);
_threads.clear();
}
template<typename FuncType>
inline
void
ThreadPool::enqueue_work(ThreadPool::PToken &ptok_,
FuncType &&func_)
{
SEMA_WAIT(_sema);
_queue.enqueue(ptok_,
std::forward<FuncType>(func_));
}
template<typename FuncType>
inline
void
ThreadPool::enqueue_work(FuncType &&func_)
{
SEMA_WAIT(_sema);
_queue.enqueue(std::forward<FuncType>(func_));
}
template<typename FuncType>
[[nodiscard]]
inline
std::future<std::invoke_result_t<FuncType>>
ThreadPool::enqueue_task(FuncType&& func_)
{
using TaskReturnType = std::invoke_result_t<FuncType>;
using Promise = std::promise<TaskReturnType>;
Promise promise;
auto future = promise.get_future();
auto work =
[promise_ = std::move(promise),
func_ = std::forward<FuncType>(func_)]() mutable
{
try
{
auto rv = func_();
promise_.set_value(std::move(rv));
}
catch(...)
{
promise_.set_exception(std::current_exception());
}
};
SEMA_WAIT(_sema);
_queue.enqueue(std::move(work));
return future;
}
inline
std::vector<pthread_t>
ThreadPool::threads() const
{
std::lock_guard<std::mutex> lg(_threads_mutex);
return _threads;
}
inline
ThreadPool::PToken
ThreadPool::ptoken()
{
return ThreadPool::PToken(_queue);
}
File diff suppressed because it is too large Load Diff
-380
View File
@@ -1,380 +0,0 @@
/*
ISC License
Copyright (c) 2021, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#pragma once
#include "kvec.h"
#include <errno.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>
#define ROUND_UP(N,S) ((((N) + (S) - 1) / (S)) * (S))
typedef kvec_t(void*) slab_kvec_t;
typedef struct mem_stack_t mem_stack_t;
struct mem_stack_t
{
mem_stack_t *next;
};
typedef struct fmp_t fmp_t;
struct fmp_t
{
mem_stack_t *objs;
slab_kvec_t slabs;
uint64_t avail_objs;
uint64_t obj_size;
uint64_t page_size;
uint64_t slab_size;
};
static
inline
uint64_t
fmp_page_size()
{
return sysconf(_SC_PAGESIZE);
}
static
inline
void
fmp_init(fmp_t *fmp_,
const uint64_t obj_size_,
const uint64_t page_multiple_)
{
kv_init(fmp_->slabs);
fmp_->objs = NULL;
fmp_->avail_objs = 0;
fmp_->obj_size = ROUND_UP(obj_size_,sizeof(void*));
fmp_->page_size = fmp_page_size();
fmp_->slab_size = (fmp_->page_size * page_multiple_);
}
static
inline
uint64_t
fmp_slab_count(fmp_t *fmp_)
{
return kv_size(fmp_->slabs);
}
static
inline
void*
fmp_slab_alloc_posix_memalign(fmp_t *fmp_)
{
int rv;
void *mem;
const size_t alignment = fmp_->page_size;
const size_t size = fmp_->slab_size;
rv = posix_memalign(&mem,alignment,size);
if(rv != 0)
return NULL;
return NULL;
}
static
inline
void*
fmp_slab_alloc_mmap(fmp_t *fmp_)
{
void *mem;
void *address = NULL;
const size_t length = fmp_->slab_size;
const int protect = PROT_READ|PROT_WRITE;
const int flags = MAP_PRIVATE|MAP_ANONYMOUS;
const int filedes = -1;
const off_t offset = 0;
mem = mmap(address,length,protect,flags,filedes,offset);
if(mem == MAP_FAILED)
return NULL;
return mem;
}
static
inline
void
fmp_slab_free_posix_memalign(fmp_t* fmp_,
void *mem_)
{
(void)fmp_;
free(mem_);
}
static
inline
void
fmp_slab_free_mmap(fmp_t* fmp_,
void *mem_)
{
void *addr = mem_;
size_t length = fmp_->slab_size;
(void)munmap(addr,length);
}
static
inline
int
fmp_slab_alloc(fmp_t *fmp_)
{
char *i;
void *mem;
mem = fmp_slab_alloc_mmap(fmp_);
if(mem == NULL)
return -ENOMEM;
kv_push(void*,fmp_->slabs,mem);
i = ((char*)mem + fmp_->slab_size - fmp_->obj_size);
while(i >= (char*)mem)
{
mem_stack_t *obj = (mem_stack_t*)i;
obj->next = fmp_->objs;
fmp_->objs = obj;
fmp_->avail_objs++;
i -= fmp_->obj_size;
}
return 0;
}
static
inline
void*
fmp_alloc(fmp_t *fmp_)
{
void *rv;
if(fmp_->objs == NULL)
fmp_slab_alloc(fmp_);
if(fmp_->objs == NULL)
return NULL;
rv = fmp_->objs;
fmp_->objs = fmp_->objs->next;
fmp_->avail_objs--;
return rv;
}
static
inline
void*
fmp_calloc(fmp_t *fmp_)
{
void *obj;
obj = fmp_alloc(fmp_);
if(obj == NULL)
return NULL;
memset(obj,0,fmp_->obj_size);
return obj;
}
static
inline
void
fmp_free(fmp_t *fmp_,
void *obj_)
{
mem_stack_t *obj = (mem_stack_t*)obj_;
obj->next = fmp_->objs;
fmp_->objs = obj;
fmp_->avail_objs++;
}
static
inline
void
fmp_clear(fmp_t *fmp_)
{
while(kv_size(fmp_->slabs))
{
void *slab = kv_pop(fmp_->slabs);
fmp_slab_free_mmap(fmp_,slab);
}
fmp_->objs = NULL;
fmp_->avail_objs = 0;
}
static
inline
void
fmp_destroy(fmp_t *fmp_)
{
fmp_clear(fmp_);
kv_destroy(fmp_->slabs);
}
static
inline
uint64_t
fmp_avail_objs(fmp_t *fmp_)
{
return fmp_->avail_objs;
}
static
inline
uint64_t
fmp_objs_per_slab(fmp_t *fmp_)
{
return (fmp_->slab_size / fmp_->obj_size);
}
static
inline
uint64_t
fmp_objs_in_slab(fmp_t *fmp_,
void *slab_)
{
char *slab;
uint64_t objs_per_slab;
uint64_t objs_in_slab;
slab = (char*)slab_;
objs_in_slab = 0;
objs_per_slab = fmp_objs_per_slab(fmp_);
for(mem_stack_t *stack = fmp_->objs; stack != NULL; stack = stack->next)
{
char *obj = (char*)stack;
if((obj >= slab) && (obj < (slab + fmp_->slab_size)))
objs_in_slab++;
if(objs_in_slab >= objs_per_slab)
break;
}
return objs_in_slab;
}
static
inline
void
fmp_remove_objs_in_slab(fmp_t *fmp_,
void *slab_)
{
char *slab;
uint64_t objs_per_slab;
uint64_t objs_in_slab;
mem_stack_t **p;
p = &fmp_->objs;
slab = (char*)slab_;
objs_in_slab = 0;
objs_per_slab = fmp_objs_per_slab(fmp_);
while((*p) != NULL)
{
char *obj = (char*)*p;
if((obj >= slab) && (obj < (slab + fmp_->slab_size)))
{
objs_in_slab++;
*p = (*p)->next;
fmp_->avail_objs--;
if(objs_in_slab >= objs_per_slab)
break;
continue;
}
p = &(*p)->next;
}
}
static
inline
int
fmp_gc_slab(fmp_t *fmp_,
uint64_t slab_idx_)
{
char *slab;
uint64_t objs_in_slab;
uint64_t objs_per_slab;
slab_idx_ = (slab_idx_ % kv_size(fmp_->slabs));
slab = (char*)kv_A(fmp_->slabs,slab_idx_);
objs_per_slab = fmp_objs_per_slab(fmp_);
objs_in_slab = fmp_objs_in_slab(fmp_,slab);
if(objs_in_slab != objs_per_slab)
return 0;
fmp_remove_objs_in_slab(fmp_,slab);
kv_delete(fmp_->slabs,slab_idx_);
fmp_slab_free_mmap(fmp_,slab);
return 1;
}
static
inline
int
fmp_gc(fmp_t *fmp_)
{
uint64_t slab_idx;
slab_idx = rand();
return fmp_gc_slab(fmp_,slab_idx);
}
static
inline
double
fmp_slab_usage_ratio(fmp_t *fmp_)
{
double avail_objs;
double objs_per_slab;
double nums_of_slabs;
avail_objs = fmp_->avail_objs;
objs_per_slab = fmp_objs_per_slab(fmp_);
nums_of_slabs = kv_size(fmp_->slabs);
return (avail_objs / (objs_per_slab * nums_of_slabs));
}
static
inline
uint64_t
fmp_total_allocated_memory(fmp_t *fmp_)
{
return (fmp_->slab_size * kv_size(fmp_->slabs));
}
-21
View File
@@ -1,21 +0,0 @@
#include "fuse_cfg.hpp"
fuse_cfg_t fuse_cfg;
bool
fuse_cfg_t::valid_uid() const
{
return (uid != FUSE_CFG_INVALID_ID);
}
bool
fuse_cfg_t::valid_gid() const
{
return (gid != FUSE_CFG_INVALID_ID);
}
bool
fuse_cfg_t::valid_umask() const
{
return (umask != FUSE_CFG_INVALID_UMASK);
}
-98
View File
@@ -1,98 +0,0 @@
/*
FUSE: Filesystem in Userspace
Copyright (C) 2001-2007 Miklos Szeredi <miklos@szeredi.hu>
This program can be distributed under the terms of the GNU LGPLv2.
See the file COPYING.LIB
*/
#pragma once
#include "fuse.h"
#include "fuse_lowlevel.h"
#include "fuse_msgbuf_t.h"
#include "extern_c.h"
struct fuse_chan;
struct fuse_ll;
struct fuse_session
{
int (*receive_buf)(struct fuse_session *se,
fuse_msgbuf_t *msgbuf);
void (*process_buf)(struct fuse_session *se,
const fuse_msgbuf_t *msgbuf);
void (*destroy)(void *data);
struct fuse_ll *f;
volatile int exited;
struct fuse_chan *ch;
};
struct fuse_notify_req
{
uint64_t unique;
void (*reply)(struct fuse_notify_req *,
fuse_req_t*,
uint64_t,
const void *);
struct fuse_notify_req *next;
struct fuse_notify_req *prev;
};
struct fuse_ll
{
struct fuse_lowlevel_ops op;
void *userdata;
uid_t owner;
struct fuse_conn_info conn;
pthread_mutex_t lock;
int got_init;
int got_destroy;
pthread_key_t pipe_key;
int broken_splice_nonblock;
uint64_t notify_ctr;
struct fuse_notify_req notify_list;
};
struct fuse_cmd
{
char *buf;
size_t buflen;
struct fuse_chan *ch;
};
EXTERN_C_BEGIN
struct fuse *fuse_new_common(struct fuse_chan *ch,
struct fuse_args *args,
const struct fuse_operations *op);
struct fuse_chan *fuse_kern_chan_new(int fd);
struct fuse_session *fuse_lowlevel_new_common(struct fuse_args *args,
const struct fuse_lowlevel_ops *op,
size_t op_size, void *userdata);
int fuse_chan_clearfd(struct fuse_chan *ch);
void fuse_kern_unmount(const char *mountpoint, int fd);
int fuse_kern_mount(const char *mountpoint, struct fuse_args *args);
int fuse_send_reply_iov_nofree(fuse_req_t req, int error, struct iovec *iov,
int count);
void fuse_free_req(fuse_req_t req);
struct fuse *fuse_setup_common(int argc,
char *argv[],
const struct fuse_operations *op,
char **mountpoint,
int *fd);
int fuse_start_thread(pthread_t *thread_id, void *(*func)(void *), void *arg);
EXTERN_C_END
-9
View File
@@ -1,9 +0,0 @@
#pragma once
#include "fuse_msgbuf_t.h"
int fuse_receive_buf(struct fuse_session *se,
fuse_msgbuf_t *msgbuf);
void fuse_process_buf(void *data,
const fuse_msgbuf_t *msgbuf,
struct fuse_chan *ch);
-240
View File
@@ -1,240 +0,0 @@
/*
ISC License
Copyright (c) 2022, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include "fuse_msgbuf.hpp"
#include "fuse.h"
#include "fuse_kernel.h"
#include <unistd.h>
#include <atomic>
#include <cstdint>
#include <cstdlib>
#include <mutex>
#include <vector>
static std::uint32_t g_PAGESIZE = 0;
static std::uint32_t g_BUFSIZE = 0;
static std::atomic<std::uint_fast64_t> g_MSGBUF_ALLOC_COUNT;
static std::mutex g_MUTEX;
static std::vector<fuse_msgbuf_t*> g_MSGBUF_STACK;
uint64_t
msgbuf_get_bufsize()
{
return g_BUFSIZE;
}
u32
msgbuf_get_pagesize()
{
return g_PAGESIZE;
}
// +1 page so that it is used for the "header" of the allocation as
// well as to allow for offseting for write requests to be page
// aligned. +1 again for fuse header as the max_pages value is for the
// body.
void
msgbuf_set_bufsize(const uint32_t size_in_pages_)
{
g_BUFSIZE = ((size_in_pages_ + 2) * g_PAGESIZE);
}
static
void
_msgbuf_page_align(fuse_msgbuf_t *msgbuf_)
{
msgbuf_->mem = (char*)msgbuf_;
msgbuf_->mem += g_PAGESIZE;
msgbuf_->size = (g_BUFSIZE - g_PAGESIZE);
}
static
void
_msgbuf_write_align(fuse_msgbuf_t *msgbuf_)
{
msgbuf_->mem = (char*)msgbuf_;
msgbuf_->mem += g_PAGESIZE;
msgbuf_->mem -= sizeof(struct fuse_in_header);
msgbuf_->mem -= sizeof(struct fuse_write_in);
msgbuf_->size = (g_BUFSIZE - g_PAGESIZE);
}
static
__attribute__((constructor))
void
_msgbuf_constructor()
{
g_PAGESIZE = sysconf(_SC_PAGESIZE);
msgbuf_set_bufsize(FUSE_DEFAULT_MAX_MAX_PAGES);
}
static
__attribute__((destructor))
void
_msgbuf_destructor()
{
}
static
void*
_page_aligned_malloc(const uint64_t size_)
{
int rv;
void *buf = NULL;
rv = posix_memalign(&buf,g_PAGESIZE,size_);
if(rv != 0)
return NULL;
return buf;
}
typedef void (*msgbuf_setup_func_t)(fuse_msgbuf_t*);
static
fuse_msgbuf_t*
_msgbuf_alloc(msgbuf_setup_func_t setup_func_)
{
fuse_msgbuf_t *msgbuf;
g_MUTEX.lock();
if(g_MSGBUF_STACK.empty())
{
g_MUTEX.unlock();
msgbuf = (fuse_msgbuf_t*)_page_aligned_malloc(g_BUFSIZE);
if(msgbuf == NULL)
return NULL;
g_MSGBUF_ALLOC_COUNT.fetch_add(1,std::memory_order_relaxed);
}
else
{
msgbuf = g_MSGBUF_STACK.back();
g_MSGBUF_STACK.pop_back();
g_MUTEX.unlock();
}
setup_func_(msgbuf);
return msgbuf;
}
// Offset the memory so write request payload will be placed on page
// boundry so O_DIRECT can work. No impact on other message types
// except for `read` which will require using `msgbuf_page_align`.
fuse_msgbuf_t*
msgbuf_alloc()
{
return _msgbuf_alloc(_msgbuf_write_align);
}
fuse_msgbuf_t*
msgbuf_alloc_page_aligned()
{
return _msgbuf_alloc(_msgbuf_page_align);
}
static
void
msgbuf_destroy(fuse_msgbuf_t *msgbuf_)
{
free(msgbuf_);
}
void
msgbuf_free(fuse_msgbuf_t *msgbuf_)
{
std::lock_guard<std::mutex> lck(g_MUTEX);
if(msgbuf_->size != (g_BUFSIZE - g_PAGESIZE))
{
msgbuf_destroy(msgbuf_);
g_MSGBUF_ALLOC_COUNT.fetch_sub(1,std::memory_order_relaxed);
return;
}
g_MSGBUF_STACK.emplace_back(msgbuf_);
}
uint64_t
msgbuf_alloc_count()
{
return g_MSGBUF_ALLOC_COUNT;
}
uint64_t
msgbuf_avail_count()
{
std::lock_guard<std::mutex> lck(g_MUTEX);
return g_MSGBUF_STACK.size();
}
void
msgbuf_gc_10percent()
{
std::vector<fuse_msgbuf_t*> togc;
{
std::size_t size;
std::size_t ten_percent;
std::lock_guard<std::mutex> lck(g_MUTEX);
size = g_MSGBUF_STACK.size();
ten_percent = (size / 10);
for(std::size_t i = 0; i < ten_percent; i++)
{
togc.push_back(g_MSGBUF_STACK.back());
g_MSGBUF_STACK.pop_back();
}
}
for(auto msgbuf : togc)
{
msgbuf_destroy(msgbuf);
g_MSGBUF_ALLOC_COUNT.fetch_sub(1,std::memory_order_relaxed);
}
}
void
msgbuf_gc()
{
std::vector<fuse_msgbuf_t*> oldstack;
{
std::lock_guard<std::mutex> lck(g_MUTEX);
oldstack.swap(g_MSGBUF_STACK);
}
for(auto msgbuf: oldstack)
{
msgbuf_destroy(msgbuf);
g_MSGBUF_ALLOC_COUNT.fetch_sub(1,std::memory_order_relaxed);
}
}
-154
View File
@@ -1,154 +0,0 @@
/*
FUSE: Filesystem in Userspace
Copyright (C) 2001-2007 Miklos Szeredi <miklos@szeredi.hu>
This program can be distributed under the terms of the GNU LGPLv2.
See the file COPYING.LIB
*/
#include "fuse_i.h"
#include "fuse_kernel.h"
#include <assert.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/uio.h>
struct fuse_chan
{
struct fuse_session *se;
int fd;
size_t bufsize;
};
struct fuse_session *
fuse_session_new(void *data,
void *receive_buf,
void *process_buf,
void *destroy)
{
struct fuse_session *se = (struct fuse_session *) malloc(sizeof(*se));
if (se == NULL) {
fprintf(stderr, "fuse: failed to allocate session\n");
return NULL;
}
memset(se, 0, sizeof(*se));
se->f = data;
se->receive_buf = receive_buf;
se->process_buf = process_buf;
se->destroy = destroy;
return se;
}
void fuse_session_add_chan(struct fuse_session *se, struct fuse_chan *ch)
{
assert(se->ch == NULL);
assert(ch->se == NULL);
se->ch = ch;
ch->se = se;
}
void fuse_session_remove_chan(struct fuse_chan *ch)
{
struct fuse_session *se = ch->se;
if (se) {
assert(se->ch == ch);
se->ch = NULL;
ch->se = NULL;
}
}
void
fuse_session_destroy(struct fuse_session *se)
{
se->destroy(se->f);
if(se->ch != NULL)
fuse_chan_destroy(se->ch);
free(se);
}
void fuse_session_reset(struct fuse_session *se)
{
se->exited = 0;
}
int
fuse_session_exited(struct fuse_session *se)
{
return se->exited;
}
void
fuse_session_exit(struct fuse_session *se_)
{
se_->exited = 1;
}
void*
fuse_session_data(struct fuse_session *se)
{
return se->f;
}
struct fuse_chan *
fuse_chan_new(int fd,
size_t bufsize)
{
struct fuse_chan *ch;
ch = (struct fuse_chan*)malloc(sizeof(*ch));
if(ch == NULL)
{
fprintf(stderr, "fuse: failed to allocate channel\n");
return NULL;
}
memset(ch, 0, sizeof(*ch));
ch->fd = fd;
ch->bufsize = bufsize;
return ch;
}
int fuse_chan_fd(struct fuse_chan *ch)
{
return ch->fd;
}
int fuse_chan_clearfd(struct fuse_chan *ch)
{
int fd = ch->fd;
ch->fd = -1;
return fd;
}
size_t fuse_chan_bufsize(struct fuse_chan *ch)
{
return ch->bufsize;
}
struct fuse_session *fuse_chan_session(struct fuse_chan *ch)
{
return ch->se;
}
void
fuse_chan_destroy(struct fuse_chan *ch)
{
int fd;
fuse_session_remove_chan(ch);
fd = fuse_chan_fd(ch);
if(fd != -1)
close(fd);
free(ch);
}
-627
View File
@@ -1,627 +0,0 @@
/* The MIT License
Copyright (c) 2008, 2009, 2011 by Attractive Chaos <attractor@live.co.uk>
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
/*
An example:
#include "khash.h"
KHASH_MAP_INIT_INT(32, char)
int main() {
int ret, is_missing;
khiter_t k;
khash_t(32) *h = kh_init(32);
k = kh_put(32, h, 5, &ret);
kh_value(h, k) = 10;
k = kh_get(32, h, 10);
is_missing = (k == kh_end(h));
k = kh_get(32, h, 5);
kh_del(32, h, k);
for (k = kh_begin(h); k != kh_end(h); ++k)
if (kh_exist(h, k)) kh_value(h, k) = 1;
kh_destroy(32, h);
return 0;
}
*/
/*
2013-05-02 (0.2.8):
* Use quadratic probing. When the capacity is power of 2, stepping function
i*(i+1)/2 guarantees to traverse each bucket. It is better than double
hashing on cache performance and is more robust than linear probing.
In theory, double hashing should be more robust than quadratic probing.
However, my implementation is probably not for large hash tables, because
the second hash function is closely tied to the first hash function,
which reduce the effectiveness of double hashing.
Reference: http://research.cs.vt.edu/AVresearch/hashing/quadratic.php
2011-12-29 (0.2.7):
* Minor code clean up; no actual effect.
2011-09-16 (0.2.6):
* The capacity is a power of 2. This seems to dramatically improve the
speed for simple keys. Thank Zilong Tan for the suggestion. Reference:
- http://code.google.com/p/ulib/
- http://nothings.org/computer/judy/
* Allow to optionally use linear probing which usually has better
performance for random input. Double hashing is still the default as it
is more robust to certain non-random input.
* Added Wang's integer hash function (not used by default). This hash
function is more robust to certain non-random input.
2011-02-14 (0.2.5):
* Allow to declare global functions.
2009-09-26 (0.2.4):
* Improve portability
2008-09-19 (0.2.3):
* Corrected the example
* Improved interfaces
2008-09-11 (0.2.2):
* Improved speed a little in kh_put()
2008-09-10 (0.2.1):
* Added kh_clear()
* Fixed a compiling error
2008-09-02 (0.2.0):
* Changed to token concatenation which increases flexibility.
2008-08-31 (0.1.2):
* Fixed a bug in kh_get(), which has not been tested previously.
2008-08-31 (0.1.1):
* Added destructor
*/
#ifndef __AC_KHASH_H
#define __AC_KHASH_H
/*!
@header
Generic hash table library.
*/
#define AC_VERSION_KHASH_H "0.2.8"
#include <stdlib.h>
#include <string.h>
#include <limits.h>
/* compiler specific configuration */
#if UINT_MAX == 0xffffffffu
typedef unsigned int khint32_t;
#elif ULONG_MAX == 0xffffffffu
typedef unsigned long khint32_t;
#endif
#if ULONG_MAX == ULLONG_MAX
typedef unsigned long khint64_t;
#else
typedef unsigned long long khint64_t;
#endif
#ifndef kh_inline
#ifdef _MSC_VER
#define kh_inline __inline
#else
#define kh_inline inline
#endif
#endif /* kh_inline */
#ifndef klib_unused
#if (defined __clang__ && __clang_major__ >= 3) || (defined __GNUC__ && __GNUC__ >= 3)
#define klib_unused __attribute__ ((__unused__))
#else
#define klib_unused
#endif
#endif /* klib_unused */
typedef khint32_t khint_t;
typedef khint_t khiter_t;
#define __ac_isempty(flag, i) ((flag[i>>4]>>((i&0xfU)<<1))&2)
#define __ac_isdel(flag, i) ((flag[i>>4]>>((i&0xfU)<<1))&1)
#define __ac_iseither(flag, i) ((flag[i>>4]>>((i&0xfU)<<1))&3)
#define __ac_set_isdel_false(flag, i) (flag[i>>4]&=~(1ul<<((i&0xfU)<<1)))
#define __ac_set_isempty_false(flag, i) (flag[i>>4]&=~(2ul<<((i&0xfU)<<1)))
#define __ac_set_isboth_false(flag, i) (flag[i>>4]&=~(3ul<<((i&0xfU)<<1)))
#define __ac_set_isdel_true(flag, i) (flag[i>>4]|=1ul<<((i&0xfU)<<1))
#define __ac_fsize(m) ((m) < 16? 1 : (m)>>4)
#ifndef kroundup32
#define kroundup32(x) (--(x), (x)|=(x)>>1, (x)|=(x)>>2, (x)|=(x)>>4, (x)|=(x)>>8, (x)|=(x)>>16, ++(x))
#endif
#ifndef kcalloc
#define kcalloc(N,Z) calloc(N,Z)
#endif
#ifndef kmalloc
#define kmalloc(Z) malloc(Z)
#endif
#ifndef krealloc
#define krealloc(P,Z) realloc(P,Z)
#endif
#ifndef kfree
#define kfree(P) free(P)
#endif
static const double __ac_HASH_UPPER = 0.77;
#define __KHASH_TYPE(name, khkey_t, khval_t) \
typedef struct kh_##name##_s { \
khint_t n_buckets, size, n_occupied, upper_bound; \
khint32_t *flags; \
khkey_t *keys; \
khval_t *vals; \
} kh_##name##_t;
#define __KHASH_PROTOTYPES(name, khkey_t, khval_t) \
extern kh_##name##_t *kh_init_##name(void); \
extern void kh_destroy_##name(kh_##name##_t *h); \
extern void kh_clear_##name(kh_##name##_t *h); \
extern khint_t kh_get_##name(const kh_##name##_t *h, khkey_t key); \
extern int kh_resize_##name(kh_##name##_t *h, khint_t new_n_buckets); \
extern khint_t kh_put_##name(kh_##name##_t *h, khkey_t key, int *ret); \
extern void kh_del_##name(kh_##name##_t *h, khint_t x);
#define __KHASH_IMPL(name, SCOPE, khkey_t, khval_t, kh_is_map, __hash_func, __hash_equal) \
SCOPE kh_##name##_t *kh_init_##name(void) { \
return (kh_##name##_t*)kcalloc(1, sizeof(kh_##name##_t)); \
} \
SCOPE void kh_destroy_##name(kh_##name##_t *h) \
{ \
if (h) { \
kfree((void *)h->keys); kfree(h->flags); \
kfree((void *)h->vals); \
kfree(h); \
} \
} \
SCOPE void kh_clear_##name(kh_##name##_t *h) \
{ \
if (h && h->flags) { \
memset(h->flags, 0xaa, __ac_fsize(h->n_buckets) * sizeof(khint32_t)); \
h->size = h->n_occupied = 0; \
} \
} \
SCOPE khint_t kh_get_##name(const kh_##name##_t *h, khkey_t key) \
{ \
if (h->n_buckets) { \
khint_t k, i, last, mask, step = 0; \
mask = h->n_buckets - 1; \
k = __hash_func(key); i = k & mask; \
last = i; \
while (!__ac_isempty(h->flags, i) && (__ac_isdel(h->flags, i) || !__hash_equal(h->keys[i], key))) { \
i = (i + (++step)) & mask; \
if (i == last) return h->n_buckets; \
} \
return __ac_iseither(h->flags, i)? h->n_buckets : i; \
} else return 0; \
} \
SCOPE int kh_resize_##name(kh_##name##_t *h, khint_t new_n_buckets) \
{ /* This function uses 0.25*n_buckets bytes of working space instead of [sizeof(key_t+val_t)+.25]*n_buckets. */ \
khint32_t *new_flags = 0; \
khint_t j = 1; \
{ \
kroundup32(new_n_buckets); \
if (new_n_buckets < 4) new_n_buckets = 4; \
if (h->size >= (khint_t)(new_n_buckets * __ac_HASH_UPPER + 0.5)) j = 0; /* requested size is too small */ \
else { /* hash table size to be changed (shrink or expand); rehash */ \
new_flags = (khint32_t*)kmalloc(__ac_fsize(new_n_buckets) * sizeof(khint32_t)); \
if (!new_flags) return -1; \
memset(new_flags, 0xaa, __ac_fsize(new_n_buckets) * sizeof(khint32_t)); \
if (h->n_buckets < new_n_buckets) { /* expand */ \
khkey_t *new_keys = (khkey_t*)krealloc((void *)h->keys, new_n_buckets * sizeof(khkey_t)); \
if (!new_keys) { kfree(new_flags); return -1; } \
h->keys = new_keys; \
if (kh_is_map) { \
khval_t *new_vals = (khval_t*)krealloc((void *)h->vals, new_n_buckets * sizeof(khval_t)); \
if (!new_vals) { kfree(new_flags); return -1; } \
h->vals = new_vals; \
} \
} /* otherwise shrink */ \
} \
} \
if (j) { /* rehashing is needed */ \
for (j = 0; j != h->n_buckets; ++j) { \
if (__ac_iseither(h->flags, j) == 0) { \
khkey_t key = h->keys[j]; \
khval_t val; \
khint_t new_mask; \
new_mask = new_n_buckets - 1; \
if (kh_is_map) val = h->vals[j]; \
__ac_set_isdel_true(h->flags, j); \
while (1) { /* kick-out process; sort of like in Cuckoo hashing */ \
khint_t k, i, step = 0; \
k = __hash_func(key); \
i = k & new_mask; \
while (!__ac_isempty(new_flags, i)) i = (i + (++step)) & new_mask; \
__ac_set_isempty_false(new_flags, i); \
if (i < h->n_buckets && __ac_iseither(h->flags, i) == 0) { /* kick out the existing element */ \
{ khkey_t tmp = h->keys[i]; h->keys[i] = key; key = tmp; } \
if (kh_is_map) { khval_t tmp = h->vals[i]; h->vals[i] = val; val = tmp; } \
__ac_set_isdel_true(h->flags, i); /* mark it as deleted in the old hash table */ \
} else { /* write the element and jump out of the loop */ \
h->keys[i] = key; \
if (kh_is_map) h->vals[i] = val; \
break; \
} \
} \
} \
} \
if (h->n_buckets > new_n_buckets) { /* shrink the hash table */ \
h->keys = (khkey_t*)krealloc((void *)h->keys, new_n_buckets * sizeof(khkey_t)); \
if (kh_is_map) h->vals = (khval_t*)krealloc((void *)h->vals, new_n_buckets * sizeof(khval_t)); \
} \
kfree(h->flags); /* free the working space */ \
h->flags = new_flags; \
h->n_buckets = new_n_buckets; \
h->n_occupied = h->size; \
h->upper_bound = (khint_t)(h->n_buckets * __ac_HASH_UPPER + 0.5); \
} \
return 0; \
} \
SCOPE khint_t kh_put_##name(kh_##name##_t *h, khkey_t key, int *ret) \
{ \
khint_t x; \
if (h->n_occupied >= h->upper_bound) { /* update the hash table */ \
if (h->n_buckets > (h->size<<1)) { \
if (kh_resize_##name(h, h->n_buckets - 1) < 0) { /* clear "deleted" elements */ \
*ret = -1; return h->n_buckets; \
} \
} else if (kh_resize_##name(h, h->n_buckets + 1) < 0) { /* expand the hash table */ \
*ret = -1; return h->n_buckets; \
} \
} /* TODO: to implement automatically shrinking; resize() already support shrinking */ \
{ \
khint_t k, i, site, last, mask = h->n_buckets - 1, step = 0; \
x = site = h->n_buckets; k = __hash_func(key); i = k & mask; \
if (__ac_isempty(h->flags, i)) x = i; /* for speed up */ \
else { \
last = i; \
while (!__ac_isempty(h->flags, i) && (__ac_isdel(h->flags, i) || !__hash_equal(h->keys[i], key))) { \
if (__ac_isdel(h->flags, i)) site = i; \
i = (i + (++step)) & mask; \
if (i == last) { x = site; break; } \
} \
if (x == h->n_buckets) { \
if (__ac_isempty(h->flags, i) && site != h->n_buckets) x = site; \
else x = i; \
} \
} \
} \
if (__ac_isempty(h->flags, x)) { /* not present at all */ \
h->keys[x] = key; \
__ac_set_isboth_false(h->flags, x); \
++h->size; ++h->n_occupied; \
*ret = 1; \
} else if (__ac_isdel(h->flags, x)) { /* deleted */ \
h->keys[x] = key; \
__ac_set_isboth_false(h->flags, x); \
++h->size; \
*ret = 2; \
} else *ret = 0; /* Don't touch h->keys[x] if present and not deleted */ \
return x; \
} \
SCOPE void kh_del_##name(kh_##name##_t *h, khint_t x) \
{ \
if (x != h->n_buckets && !__ac_iseither(h->flags, x)) { \
__ac_set_isdel_true(h->flags, x); \
--h->size; \
} \
}
#define KHASH_DECLARE(name, khkey_t, khval_t) \
__KHASH_TYPE(name, khkey_t, khval_t) \
__KHASH_PROTOTYPES(name, khkey_t, khval_t)
#define KHASH_INIT2(name, SCOPE, khkey_t, khval_t, kh_is_map, __hash_func, __hash_equal) \
__KHASH_TYPE(name, khkey_t, khval_t) \
__KHASH_IMPL(name, SCOPE, khkey_t, khval_t, kh_is_map, __hash_func, __hash_equal)
#define KHASH_INIT(name, khkey_t, khval_t, kh_is_map, __hash_func, __hash_equal) \
KHASH_INIT2(name, static kh_inline klib_unused, khkey_t, khval_t, kh_is_map, __hash_func, __hash_equal)
/* --- BEGIN OF HASH FUNCTIONS --- */
/*! @function
@abstract Integer hash function
@param key The integer [khint32_t]
@return The hash value [khint_t]
*/
#define kh_int_hash_func(key) (khint32_t)(key)
/*! @function
@abstract Integer comparison function
*/
#define kh_int_hash_equal(a, b) ((a) == (b))
/*! @function
@abstract 64-bit integer hash function
@param key The integer [khint64_t]
@return The hash value [khint_t]
*/
#define kh_int64_hash_func(key) (khint32_t)((key)>>33^(key)^(key)<<11)
/*! @function
@abstract 64-bit integer comparison function
*/
#define kh_int64_hash_equal(a, b) ((a) == (b))
/*! @function
@abstract const char* hash function
@param s Pointer to a null terminated string
@return The hash value
*/
static kh_inline khint_t __ac_X31_hash_string(const char *s)
{
khint_t h = (khint_t)*s;
if (h) for (++s ; *s; ++s) h = (h << 5) - h + (khint_t)*s;
return h;
}
/*! @function
@abstract Another interface to const char* hash function
@param key Pointer to a null terminated string [const char*]
@return The hash value [khint_t]
*/
#define kh_str_hash_func(key) __ac_X31_hash_string(key)
/*! @function
@abstract Const char* comparison function
*/
#define kh_str_hash_equal(a, b) (strcmp(a, b) == 0)
static kh_inline khint_t __ac_Wang_hash(khint_t key)
{
key += ~(key << 15);
key ^= (key >> 10);
key += (key << 3);
key ^= (key >> 6);
key += ~(key << 11);
key ^= (key >> 16);
return key;
}
#define kh_int_hash_func2(key) __ac_Wang_hash((khint_t)key)
/* --- END OF HASH FUNCTIONS --- */
/* Other convenient macros... */
/*!
@abstract Type of the hash table.
@param name Name of the hash table [symbol]
*/
#define khash_t(name) kh_##name##_t
/*! @function
@abstract Initiate a hash table.
@param name Name of the hash table [symbol]
@return Pointer to the hash table [khash_t(name)*]
*/
#define kh_init(name) kh_init_##name()
/*! @function
@abstract Destroy a hash table.
@param name Name of the hash table [symbol]
@param h Pointer to the hash table [khash_t(name)*]
*/
#define kh_destroy(name, h) kh_destroy_##name(h)
/*! @function
@abstract Reset a hash table without deallocating memory.
@param name Name of the hash table [symbol]
@param h Pointer to the hash table [khash_t(name)*]
*/
#define kh_clear(name, h) kh_clear_##name(h)
/*! @function
@abstract Resize a hash table.
@param name Name of the hash table [symbol]
@param h Pointer to the hash table [khash_t(name)*]
@param s New size [khint_t]
*/
#define kh_resize(name, h, s) kh_resize_##name(h, s)
/*! @function
@abstract Insert a key to the hash table.
@param name Name of the hash table [symbol]
@param h Pointer to the hash table [khash_t(name)*]
@param k Key [type of keys]
@param r Extra return code: -1 if the operation failed;
0 if the key is present in the hash table;
1 if the bucket is empty (never used); 2 if the element in
the bucket has been deleted [int*]
@return Iterator to the inserted element [khint_t]
*/
#define kh_put(name, h, k, r) kh_put_##name(h, k, r)
/*! @function
@abstract Retrieve a key from the hash table.
@param name Name of the hash table [symbol]
@param h Pointer to the hash table [khash_t(name)*]
@param k Key [type of keys]
@return Iterator to the found element, or kh_end(h) if the element is absent [khint_t]
*/
#define kh_get(name, h, k) kh_get_##name(h, k)
/*! @function
@abstract Remove a key from the hash table.
@param name Name of the hash table [symbol]
@param h Pointer to the hash table [khash_t(name)*]
@param k Iterator to the element to be deleted [khint_t]
*/
#define kh_del(name, h, k) kh_del_##name(h, k)
/*! @function
@abstract Test whether a bucket contains data.
@param h Pointer to the hash table [khash_t(name)*]
@param x Iterator to the bucket [khint_t]
@return 1 if containing data; 0 otherwise [int]
*/
#define kh_exist(h, x) (!__ac_iseither((h)->flags, (x)))
/*! @function
@abstract Get key given an iterator
@param h Pointer to the hash table [khash_t(name)*]
@param x Iterator to the bucket [khint_t]
@return Key [type of keys]
*/
#define kh_key(h, x) ((h)->keys[x])
/*! @function
@abstract Get value given an iterator
@param h Pointer to the hash table [khash_t(name)*]
@param x Iterator to the bucket [khint_t]
@return Value [type of values]
@discussion For hash sets, calling this results in segfault.
*/
#define kh_val(h, x) ((h)->vals[x])
/*! @function
@abstract Alias of kh_val()
*/
#define kh_value(h, x) ((h)->vals[x])
/*! @function
@abstract Get the start iterator
@param h Pointer to the hash table [khash_t(name)*]
@return The start iterator [khint_t]
*/
#define kh_begin(h) (khint_t)(0)
/*! @function
@abstract Get the end iterator
@param h Pointer to the hash table [khash_t(name)*]
@return The end iterator [khint_t]
*/
#define kh_end(h) ((h)->n_buckets)
/*! @function
@abstract Get the number of elements in the hash table
@param h Pointer to the hash table [khash_t(name)*]
@return Number of elements in the hash table [khint_t]
*/
#define kh_size(h) ((h)->size)
/*! @function
@abstract Get the number of buckets in the hash table
@param h Pointer to the hash table [khash_t(name)*]
@return Number of buckets in the hash table [khint_t]
*/
#define kh_n_buckets(h) ((h)->n_buckets)
/*! @function
@abstract Iterate over the entries in the hash table
@param h Pointer to the hash table [khash_t(name)*]
@param kvar Variable to which key will be assigned
@param vvar Variable to which value will be assigned
@param code Block of code to execute
*/
#define kh_foreach(h, kvar, vvar, code) { khint_t __i; \
for (__i = kh_begin(h); __i != kh_end(h); ++__i) { \
if (!kh_exist(h,__i)) continue; \
(kvar) = kh_key(h,__i); \
(vvar) = kh_val(h,__i); \
code; \
} }
/*! @function
@abstract Iterate over the values in the hash table
@param h Pointer to the hash table [khash_t(name)*]
@param vvar Variable to which value will be assigned
@param code Block of code to execute
*/
#define kh_foreach_value(h, vvar, code) { khint_t __i; \
for (__i = kh_begin(h); __i != kh_end(h); ++__i) { \
if (!kh_exist(h,__i)) continue; \
(vvar) = kh_val(h,__i); \
code; \
} }
/* More convenient interfaces */
/*! @function
@abstract Instantiate a hash set containing integer keys
@param name Name of the hash table [symbol]
*/
#define KHASH_SET_INIT_INT(name) \
KHASH_INIT(name, khint32_t, char, 0, kh_int_hash_func, kh_int_hash_equal)
/*! @function
@abstract Instantiate a hash map containing integer keys
@param name Name of the hash table [symbol]
@param khval_t Type of values [type]
*/
#define KHASH_MAP_INIT_INT(name, khval_t) \
KHASH_INIT(name, khint32_t, khval_t, 1, kh_int_hash_func, kh_int_hash_equal)
/*! @function
@abstract Instantiate a hash set containing 64-bit integer keys
@param name Name of the hash table [symbol]
*/
#define KHASH_SET_INIT_INT64(name) \
KHASH_INIT(name, khint64_t, char, 0, kh_int64_hash_func, kh_int64_hash_equal)
/*! @function
@abstract Instantiate a hash map containing 64-bit integer keys
@param name Name of the hash table [symbol]
@param khval_t Type of values [type]
*/
#define KHASH_MAP_INIT_INT64(name, khval_t) \
KHASH_INIT(name, khint64_t, khval_t, 1, kh_int64_hash_func, kh_int64_hash_equal)
typedef const char *kh_cstr_t;
/*! @function
@abstract Instantiate a hash map containing const char* keys
@param name Name of the hash table [symbol]
*/
#define KHASH_SET_INIT_STR(name) \
KHASH_INIT(name, kh_cstr_t, char, 0, kh_str_hash_func, kh_str_hash_equal)
/*! @function
@abstract Instantiate a hash map containing const char* keys
@param name Name of the hash table [symbol]
@param khval_t Type of values [type]
*/
#define KHASH_MAP_INIT_STR(name, khval_t) \
KHASH_INIT(name, kh_cstr_t, khval_t, 1, kh_str_hash_func, kh_str_hash_equal)
#endif /* __AC_KHASH_H */
-245
View File
@@ -1,245 +0,0 @@
/*
ISC License
Copyright (c) 2020, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#pragma once
#include "fmp.h"
#include "mutex.hpp"
#include <pthread.h>
typedef struct lfmp_t lfmp_t;
struct lfmp_t
{
fmp_t fmp;
pthread_mutex_t lock;
};
static
inline
void
lfmp_init(lfmp_t *lfmp_,
const uint64_t obj_size_,
const uint64_t page_multiple_)
{
fmp_init(&lfmp_->fmp,obj_size_,page_multiple_);
mutex_init(&lfmp_->lock);
}
static
inline
void
lfmp_lock(lfmp_t *lfmp_)
{
mutex_lock(&lfmp_->lock);
}
static
inline
void
lfmp_unlock(lfmp_t *lfmp_)
{
mutex_unlock(&lfmp_->lock);
}
static
inline
uint64_t
lfmp_slab_count(lfmp_t *lfmp_)
{
uint64_t rv;
mutex_lock(&lfmp_->lock);
rv = fmp_slab_count(&lfmp_->fmp);
mutex_unlock(&lfmp_->lock);
return rv;
}
static
inline
int
lfmp_slab_alloc(lfmp_t *lfmp_)
{
int rv;
mutex_lock(&lfmp_->lock);
rv = fmp_slab_alloc(&lfmp_->fmp);
mutex_unlock(&lfmp_->lock);
return rv;
}
static
inline
void*
lfmp_alloc(lfmp_t *lfmp_)
{
void *rv;
mutex_lock(&lfmp_->lock);
rv = fmp_alloc(&lfmp_->fmp);
mutex_unlock(&lfmp_->lock);
return rv;
}
static
inline
void*
lfmp_calloc(lfmp_t *lfmp_)
{
void *rv;
mutex_lock(&lfmp_->lock);
rv = fmp_calloc(&lfmp_->fmp);
mutex_unlock(&lfmp_->lock);
return rv;
}
static
inline
void
lfmp_free(lfmp_t *lfmp_,
void *obj_)
{
mutex_lock(&lfmp_->lock);
fmp_free(&lfmp_->fmp,obj_);
mutex_unlock(&lfmp_->lock);
}
static
inline
void
lfmp_clear(lfmp_t *lfmp_)
{
mutex_lock(&lfmp_->lock);
fmp_clear(&lfmp_->fmp);
mutex_unlock(&lfmp_->lock);
}
static
inline
void
lfmp_destroy(lfmp_t *lfmp_)
{
mutex_lock(&lfmp_->lock);
fmp_destroy(&lfmp_->fmp);
mutex_unlock(&lfmp_->lock);
mutex_destroy(&lfmp_->lock);
}
static
inline
uint64_t
lfmp_avail_objs(lfmp_t *lfmp_)
{
uint64_t rv;
mutex_lock(&lfmp_->lock);
rv = fmp_avail_objs(&lfmp_->fmp);
mutex_unlock(&lfmp_->lock);
return rv;
}
static
inline
uint64_t
lfmp_objs_in_slab(lfmp_t *lfmp_,
void *slab_)
{
uint64_t rv;
mutex_lock(&lfmp_->lock);
rv = fmp_objs_in_slab(&lfmp_->fmp,slab_);
mutex_unlock(&lfmp_->lock);
return rv;
}
static
inline
void
lfmp_remove_objs_in_slab(lfmp_t *lfmp_,
void *slab_)
{
mutex_lock(&lfmp_->lock);
fmp_remove_objs_in_slab(&lfmp_->fmp,slab_);
mutex_unlock(&lfmp_->lock);
}
static
inline
int
lfmp_gc(lfmp_t *lfmp_)
{
int rv;
mutex_lock(&lfmp_->lock);
rv = fmp_gc(&lfmp_->fmp);
mutex_unlock(&lfmp_->lock);
return rv;
}
static
inline
uint64_t
lfmp_objs_per_slab(lfmp_t *lfmp_)
{
uint64_t rv;
mutex_lock(&lfmp_->lock);
rv = fmp_objs_per_slab(&lfmp_->fmp);
mutex_unlock(&lfmp_->lock);
return rv;
}
static
inline
double
lfmp_slab_usage_ratio(lfmp_t *lfmp_)
{
double rv;
mutex_lock(&lfmp_->lock);
rv = fmp_slab_usage_ratio(&lfmp_->fmp);
mutex_unlock(&lfmp_->lock);
return rv;
}
static
inline
uint64_t
lfmp_total_allocated_memory(lfmp_t *lfmp_)
{
uint64_t rv;
mutex_lock(&lfmp_->lock);
rv = fmp_total_allocated_memory(&lfmp_->fmp);
mutex_unlock(&lfmp_->lock);
return rv;
}
-13
View File
@@ -1,13 +0,0 @@
#include <stdint.h>
#include <sys/types.h>
typedef struct lock_s lock_t;
struct lock_s
{
int type;
off_t start;
off_t end;
pid_t pid;
uint64_t owner;
lock_t *next;
};
-41
View File
@@ -1,41 +0,0 @@
#include <cstddef>
#include <memory>
#include <type_traits>
#include <utility>
namespace std
{
template<class T> struct _Unique_if
{
typedef unique_ptr<T> _Single_object;
};
template<class T> struct _Unique_if<T[]>
{
typedef unique_ptr<T[]> _Unknown_bound;
};
template<class T, size_t N> struct _Unique_if<T[N]>
{
typedef void _Known_bound;
};
template<class T, class... Args>
typename _Unique_if<T>::_Single_object
make_unique(Args&&... args)
{
return unique_ptr<T>(new T(std::forward<Args>(args)...));
}
template<class T>
typename _Unique_if<T>::_Unknown_bound
make_unique(size_t n)
{
typedef typename remove_extent<T>::type U;
return unique_ptr<T>(new U[n]());
}
template<class T, class... Args>
typename _Unique_if<T>::_Known_bound
make_unique(Args&&...) = delete;
}
-581
View File
@@ -1,581 +0,0 @@
/*
FUSE: Filesystem in Userspace
Copyright (C) 2001-2007 Miklos Szeredi <miklos@szeredi.hu>
This program can be distributed under the terms of the GNU LGPLv2.
See the file COPYING.LIB.
*/
#include "fuse_i.h"
#include "fuse_opt.h"
#include "mount_util.h"
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stddef.h>
#include <string.h>
#include <fcntl.h>
#include <errno.h>
#include <poll.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <sys/wait.h>
#include <sys/mount.h>
#ifdef __NetBSD__
#include <perfuse.h>
#define MS_RDONLY MNT_RDONLY
#define MS_NOSUID MNT_NOSUID
#define MS_NODEV MNT_NODEV
#define MS_NOEXEC MNT_NOEXEC
#define MS_SYNCHRONOUS MNT_SYNCHRONOUS
#define MS_NOATIME MNT_NOATIME
#define umount2(mnt, flags) unmount(mnt, (flags == 2) ? MNT_FORCE : 0)
#endif
#define FUSERMOUNT_PROG "fusermount"
#define FUSE_COMMFD_ENV "_FUSE_COMMFD"
#ifndef MS_DIRSYNC
#define MS_DIRSYNC 128
#endif
enum {
KEY_KERN_FLAG,
KEY_KERN_OPT,
KEY_FUSERMOUNT_OPT,
KEY_SUBTYPE_OPT,
KEY_MTAB_OPT,
KEY_RO
};
struct mount_opts {
int allow_other;
int flags;
int auto_unmount;
int blkdev;
char *fsname;
char *subtype;
char *subtype_opt;
char *mtab_opts;
char *fusermount_opts;
char *kernel_opts;
};
#define FUSE_MOUNT_OPT(t, p) { t, offsetof(struct mount_opts, p), 1 }
static
const
struct fuse_opt fuse_mount_opts[] =
{
FUSE_MOUNT_OPT("allow_other", allow_other),
FUSE_MOUNT_OPT("blkdev", blkdev),
FUSE_MOUNT_OPT("auto_unmount", auto_unmount),
FUSE_MOUNT_OPT("fsname=%s", fsname),
FUSE_MOUNT_OPT("subtype=%s", subtype),
FUSE_OPT_KEY("allow_other", KEY_KERN_OPT),
FUSE_OPT_KEY("auto_unmount", KEY_FUSERMOUNT_OPT),
FUSE_OPT_KEY("blkdev", KEY_FUSERMOUNT_OPT),
FUSE_OPT_KEY("fsname=", KEY_FUSERMOUNT_OPT),
FUSE_OPT_KEY("subtype=", KEY_SUBTYPE_OPT),
FUSE_OPT_KEY("large_read", KEY_KERN_OPT),
FUSE_OPT_KEY("blksize=", KEY_KERN_OPT),
FUSE_OPT_KEY("default_permissions", KEY_KERN_OPT),
FUSE_OPT_KEY("context=", KEY_KERN_OPT),
FUSE_OPT_KEY("fscontext=", KEY_KERN_OPT),
FUSE_OPT_KEY("defcontext=", KEY_KERN_OPT),
FUSE_OPT_KEY("rootcontext=", KEY_KERN_OPT),
FUSE_OPT_KEY("max_read=", KEY_KERN_OPT),
FUSE_OPT_KEY("max_read=", FUSE_OPT_KEY_KEEP),
FUSE_OPT_KEY("user=", KEY_MTAB_OPT),
FUSE_OPT_KEY("-r", KEY_RO),
FUSE_OPT_KEY("ro", KEY_KERN_FLAG),
FUSE_OPT_KEY("rw", KEY_KERN_FLAG),
FUSE_OPT_KEY("suid", KEY_KERN_FLAG),
FUSE_OPT_KEY("nosuid", KEY_KERN_FLAG),
FUSE_OPT_KEY("dev", KEY_KERN_FLAG),
FUSE_OPT_KEY("nodev", KEY_KERN_FLAG),
FUSE_OPT_KEY("exec", KEY_KERN_FLAG),
FUSE_OPT_KEY("noexec", KEY_KERN_FLAG),
FUSE_OPT_KEY("async", KEY_KERN_FLAG),
FUSE_OPT_KEY("sync", KEY_KERN_FLAG),
FUSE_OPT_KEY("dirsync", KEY_KERN_FLAG),
FUSE_OPT_KEY("atime", KEY_KERN_FLAG),
FUSE_OPT_KEY("noatime", KEY_KERN_FLAG),
FUSE_OPT_KEY("relatime", KEY_KERN_FLAG),
FUSE_OPT_KEY("norelatime", KEY_KERN_FLAG),
FUSE_OPT_KEY("lazytime", KEY_KERN_FLAG),
FUSE_OPT_KEY("nolazytime", KEY_KERN_FLAG),
FUSE_OPT_KEY("diratime", KEY_KERN_FLAG),
FUSE_OPT_KEY("nodiratime", KEY_KERN_FLAG),
FUSE_OPT_KEY("strictatime", KEY_KERN_FLAG),
FUSE_OPT_KEY("nostrictatime", KEY_KERN_FLAG),
FUSE_OPT_END
};
static void exec_fusermount(const char *argv[])
{
execv(FUSERMOUNT_DIR "/mergerfs-" FUSERMOUNT_PROG, (char **) argv);
execvp("mergerfs-" FUSERMOUNT_PROG, (char **) argv);
execv(FUSERMOUNT_DIR "/" FUSERMOUNT_PROG, (char **) argv);
execvp(FUSERMOUNT_PROG, (char **) argv);
}
struct mount_flags {
const char *opt;
unsigned long flag;
int on;
};
static
const
struct mount_flags
mount_flags[] =
{
{"rw", MS_RDONLY, 0},
{"ro", MS_RDONLY, 1},
{"suid", MS_NOSUID, 0},
{"nosuid", MS_NOSUID, 1},
{"dev", MS_NODEV, 0},
{"nodev", MS_NODEV, 1},
{"exec", MS_NOEXEC, 0},
{"noexec", MS_NOEXEC, 1},
{"async", MS_SYNCHRONOUS, 0},
{"sync", MS_SYNCHRONOUS, 1},
{"atime", MS_NOATIME, 0},
{"noatime", MS_NOATIME, 1},
{"relatime", MS_RELATIME, 1},
{"norelatime", MS_RELATIME, 0},
{"lazytime", MS_LAZYTIME, 1},
{"nolazytime", MS_LAZYTIME, 0},
{"diratime", MS_NODIRATIME, 0},
{"nodiratime", MS_NODIRATIME, 1},
{"strictatime", MS_STRICTATIME, 1},
{"nostrictatime",MS_STRICTATIME, 0},
#ifndef __NetBSD__
{"dirsync", MS_DIRSYNC, 1},
#endif
{NULL, 0, 0}
};
static
void
set_mount_flag(const char *s,
int *flags)
{
int i;
for (i = 0; mount_flags[i].opt != NULL; i++)
{
const char *opt = mount_flags[i].opt;
if(strcmp(opt, s) == 0)
{
if(mount_flags[i].on)
*flags |= mount_flags[i].flag;
else
*flags &= ~mount_flags[i].flag;
return;
}
}
fprintf(stderr, "fuse: internal error, can't find mount flag\n");
abort();
}
static int fuse_mount_opt_proc(void *data, const char *arg, int key,
struct fuse_args *outargs)
{
struct mount_opts *mo = data;
switch (key)
{
case KEY_RO:
arg = "ro";
/* fall through */
case KEY_KERN_FLAG:
set_mount_flag(arg, &mo->flags);
return 0;
case KEY_KERN_OPT:
return fuse_opt_add_opt(&mo->kernel_opts, arg);
case KEY_FUSERMOUNT_OPT:
return fuse_opt_add_opt_escaped(&mo->fusermount_opts, arg);
case KEY_SUBTYPE_OPT:
return fuse_opt_add_opt(&mo->subtype_opt, arg);
case KEY_MTAB_OPT:
return fuse_opt_add_opt(&mo->mtab_opts, arg);
}
return 1;
}
/* return value:
* >= 0 => fd
* -1 => error
*/
static int receive_fd(int fd)
{
struct msghdr msg;
struct iovec iov;
char buf[1];
int rv;
size_t ccmsg[CMSG_SPACE(sizeof(int)) / sizeof(size_t)];
struct cmsghdr *cmsg;
iov.iov_base = buf;
iov.iov_len = 1;
memset(&msg, 0, sizeof(msg));
msg.msg_name = 0;
msg.msg_namelen = 0;
msg.msg_iov = &iov;
msg.msg_iovlen = 1;
/* old BSD implementations should use msg_accrights instead of
* msg_control; the interface is different. */
msg.msg_control = ccmsg;
msg.msg_controllen = sizeof(ccmsg);
while(((rv = recvmsg(fd, &msg, 0)) == -1) && errno == EINTR);
if (rv == -1) {
perror("recvmsg");
return -1;
}
if(!rv) {
/* EOF */
return -1;
}
cmsg = CMSG_FIRSTHDR(&msg);
if (cmsg->cmsg_type != SCM_RIGHTS) {
fprintf(stderr, "got control message of unknown type %d\n",
cmsg->cmsg_type);
return -1;
}
return *(int*)CMSG_DATA(cmsg);
}
void fuse_kern_unmount(const char *mountpoint, int fd)
{
int res;
int pid;
if (!mountpoint)
return;
if (fd != -1) {
struct pollfd pfd;
pfd.fd = fd;
pfd.events = 0;
res = poll(&pfd, 1, 0);
/* Need to close file descriptor, otherwise synchronous umount
would recurse into filesystem, and deadlock.
Caller expects fuse_kern_unmount to close the fd, so close it
anyway. */
close(fd);
/* If file poll returns POLLERR on the device file descriptor,
then the filesystem is already unmounted */
if (res == 1 && (pfd.revents & POLLERR))
return;
}
if (geteuid() == 0) {
fuse_mnt_umount("fuse", mountpoint, mountpoint, 1);
return;
}
res = umount2(mountpoint, 2);
if (res == 0)
return;
pid = fork();
if(pid == -1)
return;
if(pid == 0) {
const char *argv[] = { FUSERMOUNT_PROG, "-u", "-q", "-z",
"--", mountpoint, NULL };
exec_fusermount(argv);
_exit(1);
}
waitpid(pid, NULL, 0);
}
static int fuse_mount_fusermount(const char *mountpoint, struct mount_opts *mo,
const char *opts, int quiet)
{
int fds[2], pid;
int res;
int rv;
if (!mountpoint) {
fprintf(stderr, "fuse: missing mountpoint parameter\n");
return -1;
}
res = socketpair(PF_UNIX, SOCK_STREAM, 0, fds);
if(res == -1) {
perror("fuse: socketpair() failed");
return -1;
}
pid = fork();
if(pid == -1) {
perror("fuse: fork() failed");
close(fds[0]);
close(fds[1]);
return -1;
}
if(pid == 0) {
char env[10];
const char *argv[32];
int a = 0;
if (quiet) {
int fd = open("/dev/null", O_RDONLY);
if (fd != -1) {
dup2(fd, 1);
dup2(fd, 2);
}
}
argv[a++] = FUSERMOUNT_PROG;
if (opts) {
argv[a++] = "-o";
argv[a++] = opts;
}
argv[a++] = "--";
argv[a++] = mountpoint;
argv[a++] = NULL;
close(fds[1]);
fcntl(fds[0], F_SETFD, 0);
snprintf(env, sizeof(env), "%i", fds[0]);
setenv(FUSE_COMMFD_ENV, env, 1);
exec_fusermount(argv);
perror("fuse: failed to exec fusermount");
_exit(1);
}
close(fds[0]);
rv = receive_fd(fds[1]);
if (!mo->auto_unmount) {
/* with auto_unmount option fusermount will not exit until
this socket is closed */
close(fds[1]);
waitpid(pid, NULL, 0); /* bury zombie */
}
return rv;
}
static int fuse_mount_sys(const char *mnt, struct mount_opts *mo,
const char *mnt_opts)
{
char tmp[128];
const char *devname = "/dev/fuse";
char *source = NULL;
char *type = NULL;
struct stat stbuf;
int fd;
int res;
if (!mnt) {
fprintf(stderr, "fuse: missing mountpoint parameter\n");
return -1;
}
res = stat(mnt, &stbuf);
if (res == -1) {
fprintf(stderr ,"fuse: failed to access mountpoint %s: %s\n",
mnt, strerror(errno));
return -1;
}
if (mo->auto_unmount) {
/* Tell the caller to fallback to fusermount because
auto-unmount does not work otherwise. */
return -2;
}
fd = open(devname, O_RDWR);
if (fd == -1) {
if (errno == ENODEV || errno == ENOENT)
fprintf(stderr, "fuse: device not found, try 'modprobe fuse' first\n");
else
fprintf(stderr, "fuse: failed to open %s: %s\n",
devname, strerror(errno));
return -1;
}
snprintf(tmp, sizeof(tmp), "fd=%i,rootmode=%o,user_id=%u,group_id=%u",
fd, stbuf.st_mode & S_IFMT, getuid(), getgid());
res = fuse_opt_add_opt(&mo->kernel_opts, tmp);
if (res == -1)
goto out_close;
source = malloc((mo->fsname ? strlen(mo->fsname) : 0) +
(mo->subtype ? strlen(mo->subtype) : 0) +
strlen(devname) + 32);
type = malloc((mo->subtype ? strlen(mo->subtype) : 0) + 32);
if (!type || !source) {
fprintf(stderr, "fuse: failed to allocate memory\n");
goto out_close;
}
strcpy(type, mo->blkdev ? "fuseblk" : "fuse");
if (mo->subtype) {
strcat(type, ".");
strcat(type, mo->subtype);
}
strcpy(source,
mo->fsname ? mo->fsname : (mo->subtype ? mo->subtype : devname));
res = mount(source, mnt, type, mo->flags, mo->kernel_opts);
if (res == -1 && errno == ENODEV && mo->subtype) {
/* Probably missing subtype support */
strcpy(type, mo->blkdev ? "fuseblk" : "fuse");
if (mo->fsname) {
if (!mo->blkdev)
sprintf(source, "%s#%s", mo->subtype,
mo->fsname);
} else {
strcpy(source, type);
}
res = mount(source, mnt, type, mo->flags, mo->kernel_opts);
}
if (res == -1) {
/*
* Maybe kernel doesn't support unprivileged mounts, in this
* case try falling back to fusermount
*/
if (errno == EPERM) {
res = -2;
} else {
int errno_save = errno;
if (mo->blkdev && errno == ENODEV &&
!fuse_mnt_check_fuseblk())
fprintf(stderr,
"fuse: 'fuseblk' support missing\n");
else
fprintf(stderr, "fuse: mount failed: %s\n",
strerror(errno_save));
}
goto out_close;
}
#ifndef __NetBSD__
#ifndef IGNORE_MTAB
if (geteuid() == 0) {
char *newmnt = fuse_mnt_resolve_path("fuse", mnt);
res = -1;
if (!newmnt)
goto out_umount;
res = fuse_mnt_add_mount("fuse", source, newmnt, type,
mnt_opts);
free(newmnt);
if (res == -1)
goto out_umount;
}
#endif /* IGNORE_MTAB */
#endif /* __NetBSD__ */
free(type);
free(source);
return fd;
out_umount:
umount2(mnt, 2); /* lazy umount */
out_close:
free(type);
free(source);
close(fd);
return res;
}
static int get_mnt_flag_opts(char **mnt_optsp, int flags)
{
int i;
if (!(flags & MS_RDONLY) && fuse_opt_add_opt(mnt_optsp, "rw") == -1)
return -1;
for (i = 0; mount_flags[i].opt != NULL; i++) {
if (mount_flags[i].on && (flags & mount_flags[i].flag) &&
fuse_opt_add_opt(mnt_optsp, mount_flags[i].opt) == -1)
return -1;
}
return 0;
}
int fuse_kern_mount(const char *mountpoint, struct fuse_args *args)
{
struct mount_opts mo;
int res = -1;
char *mnt_opts = NULL;
memset(&mo, 0, sizeof(mo));
mo.flags = MS_NOSUID | MS_NODEV;
if(args &&
fuse_opt_parse(args, &mo, fuse_mount_opts, fuse_mount_opt_proc) == -1)
return -1;
res = -1;
if (get_mnt_flag_opts(&mnt_opts, mo.flags) == -1)
goto out;
if (mo.kernel_opts && fuse_opt_add_opt(&mnt_opts, mo.kernel_opts) == -1)
goto out;
if (mo.mtab_opts && fuse_opt_add_opt(&mnt_opts, mo.mtab_opts) == -1)
goto out;
res = fuse_mount_sys(mountpoint, &mo, mnt_opts);
if (res == -2) {
if (mo.fusermount_opts &&
fuse_opt_add_opt(&mnt_opts, mo.fusermount_opts) == -1)
goto out;
if (mo.subtype) {
char *tmp_opts = NULL;
res = -1;
if (fuse_opt_add_opt(&tmp_opts, mnt_opts) == -1 ||
fuse_opt_add_opt(&tmp_opts, mo.subtype_opt) == -1) {
free(tmp_opts);
goto out;
}
res = fuse_mount_fusermount(mountpoint, &mo, tmp_opts, 1);
free(tmp_opts);
if (res == -1)
res = fuse_mount_fusermount(mountpoint, &mo,
mnt_opts, 0);
} else {
res = fuse_mount_fusermount(mountpoint, &mo, mnt_opts, 0);
}
}
out:
free(mnt_opts);
free(mo.fsname);
free(mo.subtype);
free(mo.fusermount_opts);
free(mo.subtype_opt);
free(mo.kernel_opts);
free(mo.mtab_opts);
return res;
}
-19
View File
@@ -1,19 +0,0 @@
/*
FUSE: Filesystem in Userspace
Copyright (C) 2001-2007 Miklos Szeredi <miklos@szeredi.hu>
This program can be distributed under the terms of the GNU LGPLv2.
See the file COPYING.LIB.
*/
#pragma once
#include <sys/types.h>
int fuse_mnt_add_mount(const char *progname, const char *fsname,
const char *mnt, const char *type, const char *opts);
int fuse_mnt_remove_mount(const char *progname, const char *mnt);
int fuse_mnt_umount(const char *progname, const char *abs_mnt,
const char *rel_mnt, int lazy);
char *fuse_mnt_resolve_path(const char *progname, const char *orig);
int fuse_mnt_check_fuseblk(void);
-60
View File
@@ -1,60 +0,0 @@
#include "node.hpp"
#include "lfmp.h"
static lfmp_t g_NODE_FMP;
static
__attribute__((constructor))
void
node_constructor()
{
lfmp_init(&g_NODE_FMP,sizeof(node_t),256);
}
static
__attribute__((destructor))
void
node_destructor()
{
lfmp_destroy(&g_NODE_FMP);
}
node_t*
node_alloc()
{
return (node_t*)lfmp_calloc(&g_NODE_FMP);
}
void
node_free(node_t *node_)
{
lfmp_free(&g_NODE_FMP,node_);
}
int
node_gc1()
{
return lfmp_gc(&g_NODE_FMP);
}
void
node_gc()
{
int rv;
int fails;
fails = 0;
do
{
rv = node_gc1();
if(rv == 0)
fails++;
} while(rv || (fails < 3));
}
lfmp_t*
node_lfmp()
{
return &g_NODE_FMP;
}
-31
View File
@@ -1,31 +0,0 @@
#pragma once
#include "lock.h"
#include "lfmp.h"
typedef struct node_s node_t;
struct node_s
{
node_t *name_next;
node_t *id_next;
uint64_t nodeid;
char *name;
node_t *parent;
uint64_t nlookup;
uint32_t refctr;
uint32_t open_count;
int32_t treelock;
lock_t *locks;
uint32_t stat_crc32b;
uint8_t is_stat_cache_valid:1;
};
node_t *node_alloc();
void node_free(node_t*);
int node_gc1();
void node_gc();
lfmp_t *node_lfmp();
-28
View File
@@ -1,28 +0,0 @@
#pragma once
#include "cpu.hpp"
#include <string>
namespace PinThreads
{
void R1L(const CPU::ThreadIdVec threads);
void R1P(const CPU::ThreadIdVec threads);
void RP1L(const CPU::ThreadIdVec read_threads,
const CPU::ThreadIdVec process_threads);
void RP1P(const CPU::ThreadIdVec read_threads,
const CPU::ThreadIdVec process_threads);
void R1LP1L(const CPU::ThreadIdVec read_threads,
const CPU::ThreadIdVec process_threads);
void R1PP1P(const CPU::ThreadIdVec read_threads,
const CPU::ThreadIdVec process_threads);
void RPSL(const CPU::ThreadIdVec read_threads,
const CPU::ThreadIdVec process_threads);
void RPSP(const CPU::ThreadIdVec read_threads,
const CPU::ThreadIdVec process_threads);
void R1PPSP(const CPU::ThreadIdVec read_threads,
const CPU::ThreadIdVec process_threads);
void pin(const CPU::ThreadIdVec read_threads,
const CPU::ThreadIdVec process_threads,
const std::string type);
}
-236
View File
@@ -1,236 +0,0 @@
/*
FUSE: Filesystem in Userspace
Copyright (C) 2001-2007 Miklos Szeredi <miklos@szeredi.hu>
This program can be distributed under the terms of the GNU GPL.
See the file COPYING.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
static char *progname;
static char *xstrdup(const char *s)
{
char *t = strdup(s);
if (!t) {
fprintf(stderr, "%s: failed to allocate memory\n", progname);
exit(1);
}
return t;
}
static void *xrealloc(void *oldptr, size_t size)
{
void *ptr = realloc(oldptr, size);
if (!ptr) {
fprintf(stderr, "%s: failed to allocate memory\n", progname);
exit(1);
}
return ptr;
}
static void add_arg(char **cmdp, const char *opt)
{
size_t optlen = strlen(opt);
size_t cmdlen = *cmdp ? strlen(*cmdp) : 0;
char *cmd = xrealloc(*cmdp, cmdlen + optlen * 4 + 4);
char *s;
s = cmd + cmdlen;
if (*cmdp)
*s++ = ' ';
*s++ = '\'';
for (; *opt; opt++) {
if (*opt == '\'') {
*s++ = '\'';
*s++ = '\\';
*s++ = '\'';
*s++ = '\'';
} else
*s++ = *opt;
}
*s++ = '\'';
*s = '\0';
*cmdp = cmd;
}
static char *add_option(const char *opt, char *options)
{
int oldlen = options ? strlen(options) : 0;
options = xrealloc(options, oldlen + 1 + strlen(opt) + 1);
if (!oldlen)
strcpy(options, opt);
else {
strcat(options, ",");
strcat(options, opt);
}
return options;
}
int main(int argc, char *argv[])
{
char *type = NULL;
char *source;
const char *mountpoint;
char *basename;
char *options = NULL;
char *command = NULL;
char *setuid = NULL;
int i;
int dev = 1;
int suid = 1;
progname = argv[0];
basename = strrchr(argv[0], '/');
if (basename)
basename++;
else
basename = argv[0];
type = "mergerfs";
if (strncmp(basename, "mount.fuse.", 11) == 0)
type = basename + 11;
if (strncmp(basename, "mount.fuseblk.", 14) == 0)
type = basename + 14;
if (type && !type[0])
type = NULL;
if (argc < 3) {
fprintf(stderr,
"usage: %s %s destination [-t type] [-o opt[,opts...]]\n",
progname, type ? "source" : "type#[source]");
exit(1);
}
source = argv[1];
if (!source[0])
source = NULL;
mountpoint = argv[2];
for (i = 3; i < argc; i++) {
if (strcmp(argv[i], "-v") == 0) {
continue;
} else if (strcmp(argv[i], "-t") == 0) {
i++;
if (i == argc) {
fprintf(stderr,
"%s: missing argument to option '-t'\n",
progname);
exit(1);
}
type = argv[i];
if (strncmp(type, "fuse.", 5) == 0)
type += 5;
else if (strncmp(type, "fuseblk.", 8) == 0)
type += 8;
if (!type[0]) {
fprintf(stderr,
"%s: empty type given as argument to option '-t'\n",
progname);
exit(1);
}
} else if (strcmp(argv[i], "-o") == 0) {
char *opts;
char *opt;
i++;
if (i == argc)
break;
opts = xstrdup(argv[i]);
opt = strtok(opts, ",");
while (opt)
{
int j;
int ignore = 0;
const char *ignore_opts[] =
{
"",
"nofail",
"user",
"nouser",
"users",
"auto",
"noauto",
"_netdev",
NULL
};
if (strncmp(opt, "setuid=", 7) == 0) {
setuid = xstrdup(opt + 7);
ignore = 1;
}
for (j = 0; ignore_opts[j]; j++)
if (strcmp(opt, ignore_opts[j]) == 0)
ignore = 1;
if (!ignore) {
if (strcmp(opt, "nodev") == 0)
dev = 0;
else if (strcmp(opt, "nosuid") == 0)
suid = 0;
options = add_option(opt, options);
}
opt = strtok(NULL, ",");
}
}
}
if (dev)
options = add_option("dev", options);
if (suid)
options = add_option("suid", options);
if (!type) {
if (source) {
type = xstrdup(source);
source = strchr(type, '#');
if (source)
*source++ = '\0';
if (!type[0]) {
fprintf(stderr, "%s: empty filesystem type\n",
progname);
exit(1);
}
} else {
fprintf(stderr, "%s: empty source\n", progname);
exit(1);
}
}
add_arg(&command, type);
if (source)
add_arg(&command, source);
add_arg(&command, mountpoint);
if (options) {
add_arg(&command, "-o");
add_arg(&command, options);
}
if (setuid && setuid[0]) {
char *sucommand = command;
command = NULL;
add_arg(&command, "su");
add_arg(&command, "-");
add_arg(&command, setuid);
add_arg(&command, "-c");
add_arg(&command, sucommand);
} else if (!getenv("HOME")) {
/* Hack to make filesystems work in the boot environment */
setenv("HOME", "/root", 0);
}
execl("/bin/sh", "/bin/sh", "-c", command, NULL);
fprintf(stderr, "%s: failed to execute /bin/sh: %s\n", progname,
strerror(errno));
return 1;
}
-330
View File
@@ -1,330 +0,0 @@
/*
FUSE: Filesystem in Userspace
Copyright (C) 2001-2007 Miklos Szeredi <miklos@szeredi.hu>
This program can be distributed under the terms of the GNU LGPLv2.
See the file COPYING.LIB.
*/
#include "mount_util.h"
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <paths.h>
#ifndef __NetBSD__
#include <mntent.h>
#endif
#include <sys/stat.h>
#include <sys/wait.h>
#include <sys/mount.h>
#include <sys/param.h>
#ifdef __NetBSD__
#define umount2(mnt, flags) unmount(mnt, (flags == 2) ? MNT_FORCE : 0)
#define mtab_needs_update(mnt) 0
#else
static int mtab_needs_update(const char *mnt)
{
int res;
struct stat stbuf;
/* If mtab is within new mount, don't touch it */
if (strncmp(mnt, _PATH_MOUNTED, strlen(mnt)) == 0 &&
_PATH_MOUNTED[strlen(mnt)] == '/')
return 0;
/*
* Skip mtab update if /etc/mtab:
*
* - doesn't exist,
* - is a symlink,
* - is on a read-only filesystem.
*/
res = lstat(_PATH_MOUNTED, &stbuf);
if (res == -1) {
if (errno == ENOENT)
return 0;
} else {
uid_t ruid;
int err;
if (S_ISLNK(stbuf.st_mode))
return 0;
ruid = getuid();
if (ruid != 0)
setreuid(0, -1);
res = access(_PATH_MOUNTED, W_OK);
err = (res == -1) ? errno : 0;
if (ruid != 0)
setreuid(ruid, -1);
if (err == EROFS)
return 0;
}
return 1;
}
#endif /* __NetBSD__ */
static int add_mount(const char *progname, const char *fsname,
const char *mnt, const char *type, const char *opts)
{
int res;
int status;
sigset_t blockmask;
sigset_t oldmask;
sigemptyset(&blockmask);
sigaddset(&blockmask, SIGCHLD);
res = sigprocmask(SIG_BLOCK, &blockmask, &oldmask);
if (res == -1) {
fprintf(stderr, "%s: sigprocmask: %s\n", progname, strerror(errno));
return -1;
}
res = fork();
if (res == -1) {
fprintf(stderr, "%s: fork: %s\n", progname, strerror(errno));
goto out_restore;
}
if (res == 0) {
char *env = NULL;
sigprocmask(SIG_SETMASK, &oldmask, NULL);
setuid(geteuid());
execle("/bin/mount", "/bin/mount", "--no-canonicalize", "-i",
"-f", "-t", type, "-o", opts, fsname, mnt, NULL, &env);
fprintf(stderr, "%s: failed to execute /bin/mount: %s\n",
progname, strerror(errno));
exit(1);
}
res = waitpid(res, &status, 0);
if (res == -1)
fprintf(stderr, "%s: waitpid: %s\n", progname, strerror(errno));
if (status != 0)
res = -1;
out_restore:
sigprocmask(SIG_SETMASK, &oldmask, NULL);
return res;
}
int fuse_mnt_add_mount(const char *progname, const char *fsname,
const char *mnt, const char *type, const char *opts)
{
if (!mtab_needs_update(mnt))
return 0;
return add_mount(progname, fsname, mnt, type, opts);
}
static int exec_umount(const char *progname, const char *rel_mnt, int lazy)
{
int res;
int status;
sigset_t blockmask;
sigset_t oldmask;
sigemptyset(&blockmask);
sigaddset(&blockmask, SIGCHLD);
res = sigprocmask(SIG_BLOCK, &blockmask, &oldmask);
if (res == -1) {
fprintf(stderr, "%s: sigprocmask: %s\n", progname, strerror(errno));
return -1;
}
res = fork();
if (res == -1) {
fprintf(stderr, "%s: fork: %s\n", progname, strerror(errno));
goto out_restore;
}
if (res == 0) {
char *env = NULL;
sigprocmask(SIG_SETMASK, &oldmask, NULL);
setuid(geteuid());
if (lazy) {
execle("/bin/umount", "/bin/umount", "-i", rel_mnt,
"-l", NULL, &env);
} else {
execle("/bin/umount", "/bin/umount", "-i", rel_mnt,
NULL, &env);
}
fprintf(stderr, "%s: failed to execute /bin/umount: %s\n",
progname, strerror(errno));
exit(1);
}
res = waitpid(res, &status, 0);
if (res == -1)
fprintf(stderr, "%s: waitpid: %s\n", progname, strerror(errno));
if (status != 0) {
res = -1;
}
out_restore:
sigprocmask(SIG_SETMASK, &oldmask, NULL);
return res;
}
int fuse_mnt_umount(const char *progname, const char *abs_mnt,
const char *rel_mnt, int lazy)
{
int res;
if (!mtab_needs_update(abs_mnt)) {
res = umount2(rel_mnt, lazy ? 2 : 0);
if (res == -1)
fprintf(stderr, "%s: failed to unmount %s: %s\n",
progname, abs_mnt, strerror(errno));
return res;
}
return exec_umount(progname, rel_mnt, lazy);
}
static int remove_mount(const char *progname, const char *mnt)
{
int res;
int status;
sigset_t blockmask;
sigset_t oldmask;
sigemptyset(&blockmask);
sigaddset(&blockmask, SIGCHLD);
res = sigprocmask(SIG_BLOCK, &blockmask, &oldmask);
if (res == -1) {
fprintf(stderr, "%s: sigprocmask: %s\n", progname, strerror(errno));
return -1;
}
res = fork();
if (res == -1) {
fprintf(stderr, "%s: fork: %s\n", progname, strerror(errno));
goto out_restore;
}
if (res == 0) {
char *env = NULL;
sigprocmask(SIG_SETMASK, &oldmask, NULL);
setuid(geteuid());
execle("/bin/umount", "/bin/umount", "--no-canonicalize", "-i",
"--fake", mnt, NULL, &env);
fprintf(stderr, "%s: failed to execute /bin/umount: %s\n",
progname, strerror(errno));
exit(1);
}
res = waitpid(res, &status, 0);
if (res == -1)
fprintf(stderr, "%s: waitpid: %s\n", progname, strerror(errno));
if (status != 0)
res = -1;
out_restore:
sigprocmask(SIG_SETMASK, &oldmask, NULL);
return res;
}
int fuse_mnt_remove_mount(const char *progname, const char *mnt)
{
if (!mtab_needs_update(mnt))
return 0;
return remove_mount(progname, mnt);
}
char *fuse_mnt_resolve_path(const char *progname, const char *orig)
{
char buf[PATH_MAX];
char *copy;
char *dst;
char *end;
char *lastcomp;
const char *toresolv;
if (!orig[0]) {
fprintf(stderr, "%s: invalid mountpoint '%s'\n", progname,
orig);
return NULL;
}
copy = strdup(orig);
if (copy == NULL) {
fprintf(stderr, "%s: failed to allocate memory\n", progname);
return NULL;
}
toresolv = copy;
lastcomp = NULL;
for (end = copy + strlen(copy) - 1; end > copy && *end == '/'; end --);
if (end[0] != '/') {
char *tmp;
end[1] = '\0';
tmp = strrchr(copy, '/');
if (tmp == NULL) {
lastcomp = copy;
toresolv = ".";
} else {
lastcomp = tmp + 1;
if (tmp == copy)
toresolv = "/";
}
if (strcmp(lastcomp, ".") == 0 || strcmp(lastcomp, "..") == 0) {
lastcomp = NULL;
toresolv = copy;
}
else if (tmp)
tmp[0] = '\0';
}
if (realpath(toresolv, buf) == NULL) {
fprintf(stderr, "%s: bad mount point %s: %s\n", progname, orig,
strerror(errno));
free(copy);
return NULL;
}
if (lastcomp == NULL)
dst = strdup(buf);
else {
dst = (char *) malloc(strlen(buf) + 1 + strlen(lastcomp) + 1);
if (dst) {
unsigned buflen = strlen(buf);
if (buflen && buf[buflen-1] == '/')
sprintf(dst, "%s%s", buf, lastcomp);
else
sprintf(dst, "%s/%s", buf, lastcomp);
}
}
free(copy);
if (dst == NULL)
fprintf(stderr, "%s: failed to allocate memory\n", progname);
return dst;
}
int fuse_mnt_check_fuseblk(void)
{
char buf[256];
FILE *f = fopen("/proc/filesystems", "r");
if (!f)
return 1;
while (fgets(buf, sizeof(buf), f))
if (strstr(buf, "fuseblk\n")) {
fclose(f);
return 1;
}
fclose(f);
return 0;
}
+1 -1
View File
@@ -4,4 +4,4 @@ set -e
python3 -m venv venv
. ./venv/bin/activate
pip3 install --no-cache-dir mkdocs mkdocs-material pymdown-extensions mike
pip3 install --no-cache-dir 'mkdocs<2.0' mkdocs-material pymdown-extensions mike
+5 -5
View File
@@ -58,13 +58,13 @@ read/writes. Contact the software author or worse case the mergerfs
author so it may be investigated further.
### write benchmark
### Write Benchmark
```
$ dd if=/dev/zero of=/mnt/mergerfs/16GB.file bs=1M count=16384 oflag=nocache conv=fdatasync status=progress
```
### read benchmark
### Read Benchmark
```
$ dd if=/mnt/mergerfs/16GB.file of=/dev/null bs=1M iflag=nocache conv=fdatasync status=progress
@@ -79,21 +79,21 @@ Below are some basic tests but also try setting `--direct=0`, changing
the `ioengine`, or playing with other options.
#### write
#### Write
```
$ fio --name=writetest --filename=/mnt/mergerfs/16GB.file --size=16Gb --rw=write --bs=1M --direct=1 --numjobs=1 --iodepth=8 --group_reporting --runtime=60 --startdelay=0 --ioengine=psync
```
#### read
#### Read
```
$ fio --name=readtest --filename=/mnt/mergerfs/16GB.file --size=16Gb --rw=read --bs=1M --direct=1 --numjobs=1 --iodepth=8 --group_reporting --runtime=60 --startdelay=0 --ioengine=psync
```
### other benchmarks
### Other Benchmarks
If you are attempting to benchmark other behaviors you must ensure you
clear kernel caches before runs. In fact it would be a good deal to
@@ -38,6 +38,32 @@ These files and xattrs are used to help `mergerfs` know whether or not
a directory happens to be a branch or is the underlying directory
mount point.
## Paths which appear after startup
The [`branches` resolution process](branches.md#resolution) runs before
this option starts waiting. A missing literal path is retained as a
branch, so a configuration such as `/mnt/hdd/disk0/foobar` can work
when `/mnt/hdd/disk0` is mounted asynchronously and `foobar` only
becomes visible afterward, provided the path then meets one of the
"mounted" conditions above.
For each branch which is not ready, mergerfs first runs `mount` once
with the full branch path as the target and then polls that same path
until the timeout expires. It does not create the path, try to mount
its parent, or rerun `mount`. In the example above, the initial
`mount /mnt/hdd/disk0/foobar` may fail while the path is absent. If
another service mounts `/mnt/hdd/disk0` and
`/mnt/hdd/disk0/foobar` appears and meets one of the "mounted"
conditions before the deadline, mergerfs detects it during polling
and continues.
Globs are different. If `/mnt/hdd/disk0/*` has no matches during
branch resolution, the literal path `/mnt/hdd/disk0/*` is retained.
The mount wait does not expand it again after `/mnt/hdd/disk0`
becomes available. Ensure the parent filesystem is mounted before
mergerfs, configure a literal branch path, or update `branches`
through the runtime API once the paths exist.
**NOTE:** If on a `systemd` based system and using `fstab` it is a
good idea to set the mount option
+32 -5
View File
@@ -14,7 +14,7 @@ Branches currently have two options which can be set. A
included in a policy calculation and a individual
[minfreespace](#minfreespace) value. The values are set by
prepending an `=` at the end of a branch designation and using commas
as delimiters. Example: `/mnt/drive=RW,1234`
as delimiters. Example: `/mnt/hdd/disk0=RW,1234`
### branch mode
@@ -34,6 +34,33 @@ Same purpose and syntax as the [global option](minfreespace.md) but
specific to the branch. Defaults to the global value.
### resolution
Each colon-delimited branch designation is resolved independently when
the `branches` option is parsed, both at startup and when it is updated
through the runtime API:
1. The path portion is expanded as a glob.
2. If the glob matches one or more directories, each match becomes a
branch.
3. If the glob has no matches, mergerfs logs a notice and keeps the
configured path unchanged as a single, literal branch.
4. mergerfs resolves each resulting path with `realpath`. If that
succeeds, the canonical path is stored. If it fails, typically
because the path does not exist yet, the unchanged path is stored.
5. A path which exists but is not a directory is logged and skipped. A
path which does not exist is logged but retained.
Consequently, a literal branch such as `/mnt/hdd/disk0/foobar` may be
configured before it exists and can become usable after
`/mnt/hdd/disk0` is mounted. Branch resolution is not repeated in the
background, however. If `/mnt/hdd/disk0/*` has no matches when it is
resolved, mergerfs stores the literal path `/mnt/hdd/disk0/*`;
directories which match it later are not discovered automatically.
Update `branches` through the runtime API or restart mergerfs to
resolve the glob again.
### globbing
To make it easier to include multiple branches mergerfs supports
@@ -42,18 +69,18 @@ MUST be escaped when using via the shell else the shell itself will
apply the glob itself.**
```
# mergerfs /mnt/hdd\*:/mnt/ssd /media
# mergerfs /mnt/hdd/\*:/mnt/ssd/\* /media
```
The above line will use all directories in /mnt prefixed with **hdd**
as well as **ssd**.
The above line will use all directories within `/mnt/hdd` and
`/mnt/ssd`.
To have the pool mounted at boot or otherwise accessible from related
tools use `/etc/fstab`.
```
# <file system> <mount point> <type> <options> <dump> <pass>
/mnt/hdd*:/mnt/ssd /media mergerfs minfreespace=16G 0 0
/mnt/hdd/*:/mnt/ssd/* /media mergerfs minfreespace=16G 0 0
```
**NOTE:** The globbing is done at mount or when updated using the
+4 -1
View File
@@ -7,14 +7,17 @@ function or have been replaced. **They should not be used.**
automatically if running as root.
* **async_read**: Use `async-read=true`.
* **atomic_o_trunc**: Does nothing.
* **big_writes**: Does nothing.
* **attr_timeout**: Use [cache.attr](cache.md).
* **auto_cache**: Use [cache.files=auto-full](cache.md).
* **big_writes**: Does nothing.
* **direct_io**: Use [cache.files=off](cache.md).
* **entry_timeout**: Use [cache.entry](cache.md).
* **hard_remove**: Does nothing.
* **kernel_cache**: Use [cache.files=full](cache.md).
* **negative_entry**: Use [cache.negative-entry](cache.md).
* **remember**: Use `never-forget-nodes`.
* **remember-nodes**: Use `never-forget-nodes`.
* **noforget**: Use `never-forget-nodes`.
* **nonempty**: Does nothing.
* **splice_move**: Does nothing.
* **splice_read**: Does nothing.
+9 -6
View File
@@ -213,14 +213,17 @@ config file.
* **passthrough.max-stack-depth=INT**: Set to `1` another filesystem
can be stacked on mergerfs. Set to `2` to have mergerfs stacked over
another filesystem. (default: 1)
* **remember-nodes=INT**: The number of seconds to keep the internal
representation of a file once the OS tells mergerfs it is no longer
needed. Really only needed for [exporting mergerfs via
NFS](../remote_filesystems.md) (default: 0)
* **noforget**: Effectively sets `remember-nodes` to infinity.
* **never-forget-nodes=BOOL**: Keep the internal representation of
a file after the OS tells mergerfs it is no longer needed, until the
node is unlinked, removed, or replaced. Really only needed for
[exporting mergerfs via NFS](../remote_filesystems.md). Formerly
`noforget`. (default: false)
* **allow-idmap=BOOL**: Enables idmap (identity mapping). (default:
true)
* **debug**: Enable debugging information. At the moment limited to a
FUSE message trace saved to `log.file`. (default: false)
* **log.file**: The file path for the FUSE message trace. Empty string
will set to stderr. (default: stderr)
**NOTE:** Options are evaluated in the order listed so if the options
are **func.rmdir=rand,category.action=ff** the **action** category
+2 -1
View File
@@ -2,7 +2,8 @@
If using path preservation and a `rename` fails with `EXDEV`:
1. Move file from `/branch/a/b/c` to `/branch/.mergerfs_rename_exdev/a/b/c`.
1. Move file from `/mnt/hdd/disk0/a/b/c` to
`/mnt/hdd/disk0/.mergerfs_rename_exdev/a/b/c`.
2. symlink the rename's `newpath` to the moved file.
The `target` value is determined by the value of `rename-exdev`.
+12 -5
View File
@@ -88,11 +88,18 @@ filesystems. Mainly `ext4`.
## Logging
Filesystems, and therefore mergerfs, are doing lots of small actions
at high speed. It simply isn't reasonable to log all the actions of
the system. That said: certain details are logged at startup and when
performing maintenance tasks. These are logged via `syslog` and on
`systemd` based systems can be viewed by running
Filesystems, and therefore mergerfs, are doing **lots** of small
actions at high speed. It simply isn't reasonable to log all the
actions of the system all the time. However, there is a debug mode
which can be toggled on as needed, even at runtime, which will record
some information to help with debugging. The main thing it will record
is a trace of all FUSE messages to the location defined by the
`log.file` option. See [config section](config/options.md) for more
details.
That said even with debug mode disabled mergerfs will log certain
details via `syslog` which on `systemd` based systems can be viewed by
running:
```
journalctl -t mergerfs
@@ -67,7 +67,38 @@ Yes. They are completely unrelated pieces of software that just happen
to work well together.
## Can I use mergerfs without nonraid? nonraid without mergerfs?
## How does mergerfs interact with SnapRAID recovery behavior?
SnapRAID does not choose parity layout based on directory structure or
relative path. As each file is discovered, SnapRAID fills parity space
for that file based on its size using the next available parity area.
This makes the logical parity layout depend on discovery order, file
size, and how much data already exists on each filesystem at that
time. Filesystems with similar used space, regardless of total
capacity, are more likely to place newly added files into similar
parity positions.
Because of that, if files are changed or removed across multiple
branches and those files happen to overlap in parity space, recovery
of those deleted files may be less reliable. This is not a general
concern, is not unique to mergerfs usage, and it does not affect
filesystem failure recovery. It is specifically a concern when trying
to recover from accidental deletion of files spread across multiple
filesystems.
In practice this mostly matters when the filesystems have very similar
used space, the files are created across those filesystems around the
same time, and you later want SnapRAID to recover an accidental
deletion spanning multiple filesystems. Otherwise this is usually not
something to worry about.
If, however, this is a concern it can be beneficial to keep files on
as few filesystems as possible or otherwise keep related files
together. See the [collocation
section.](configuration_and_policies.md#how-can-i-ensure-files-are-collocated-on-the-same-branch)
## Can I use mergerfs without NonRAID? NonRAID without mergerfs?
[https://github.com/qvr/nonraid](https://github.com/qvr/nonraid)
@@ -75,6 +106,14 @@ Yes. They are completely unrelated pieces of software that just happen
to work well together.
### How does mergerfs interact with NonRAID?
There are no known special interactions or concerns between mergerfs
and NonRAID. They are separate tools and mergerfs has no known
behavior which requires any special configuration or caveats when used
with NonRAID.
## Does mergerfs support CoW / copy-on-write / writes to read-only filesystems?
Not in the sense of a filesystem like BTRFS or ZFS nor in the
@@ -110,13 +149,79 @@ It is generally recommended to not use user namespacing / id mapping
given the complication it introduces.
## Does mergerfs work with SELinux relabeling?
FUSE filesystems, including mergerfs, do not support per-inode
SELinux labels (see [Red Hat bug
1272868](https://bugzilla.redhat.com/show_bug.cgi?id=1272868) and the
[upstream fuse-devel
thread](https://sourceforge.net/p/fuse/mailman/fuse-devel/thread/CAN9vWDLGA4%3Dg3peKVWkX7gLRiVUUvJAQx9kiXHgqdeX6eh0MYA%40mail.gmail.com/)).
When a container runtime such as Podman or Docker is asked to relabel
a bind mount with the `:z` or `:Z` mount option, it first probes the
source for `security.selinux` xattr support. On a FUSE path that
probe returns `EOPNOTSUPP` and the runtime skips relabeling entirely.
Files inside the mount keep the `fusefs_t` type assigned by the
kernel. No tree walk is performed and no journal noise is produced at
default verbosity. The runtime logs `Labeling not supported on
"<path>"` only at debug level (for example
`podman --log-level=debug`). The flag is therefore a no-op against a
mergerfs mount; not harmful, but misleading.
Containers can still read and write a mergerfs pool without `:z`
because two independent access checks must succeed and both can be
satisfied without any relabel:
* **SELinux:** on Fedora, RHEL / CentOS Stream, and SUSE the shipped
[container-selinux](https://github.com/containers/container-selinux)
policy already permits the `container_t` process type to manage
files of type `fusefs_t`. See `container_selinux(8)`.
* **DAC:** the in-container UID and GID must still be able to read
and write the files on the underlying branches. With rootless
Podman this is most easily arranged by aligning the container UID
with the host owner of the branches (for example
`--userns=keep-id`, or Quadlet `UserNS=keep-id`).
The recommendation is to omit `:z` and `:Z` on bind mounts whose
source is a mergerfs (or any other FUSE) path, and to keep them on
bind mounts whose source is a regular local filesystem (ext4, btrfs,
xfs, etc.) where relabeling does real work. On distributions whose
container-selinux policy does not already include the `fusefs_t`
allow rule, the targeted fix is to set a single label at mount time
by adding `context="system_u:object_r:container_file_t:s0"` to the
mergerfs mount options (for example in `/etc/fstab`). Files inside
the pool then appear as `container_file_t`, which container-selinux
permits `container_t` to manage everywhere. See [Red Hat's mounting
file systems
guide](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-security-enhanced_linux-working_with_selinux-mounting_file_systems).
This is preferred over the much broader
`--security-opt label=disable`, which removes SELinux confinement
from the container as a whole. The trade-off of `context=` is that
host-side processes touching the mount (rsync, backup tools, web
UIs, cron jobs) see `container_file_t` rather than `fusefs_t` and
may require a small SELinux policy adjustment.
To verify on your own system:
```
podman --log-level=debug run --rm \
-v /path/to/mergerfs:/data:z \
alpine true 2>&1 | grep -iE 'relabel|setfilecon|Labeling'
```
Expected output is a single line of the form
`Labeling not supported on "/path/to/mergerfs"` and nothing in
`ausearch -m AVC,SELINUX_ERR` for the mount.
## Can mergerfs be used with NFS root squash?
If mergerfs is pooling a NFS mount then root squash should be disabled
If mergerfs is pooling a NFS mount then root squash must be disabled
as mergerfs needs to be able to have elevated privileges to do what it
does.
does (set permissions, ownership, etc.)
If you are exporting mergerfs over NFS then it is not really necessary.
If you are exporting mergerfs over NFS it is not strictly necessary to
disable root squashing but 3rd party software running as root may
fail.
See the [section on remote filesystems.](../remote_filesystems.md)
@@ -18,6 +18,12 @@ mergerfs [does not impact](usage_and_functionality.md) the underlying
filesystems and can be added or removed without any impact it is
extremely easy to test and experiment with different settings.
[mergerfs-webui](../tooling.md#mergerfs-webui) provides a browser-based
way to inspect and change settings, branches, and policies on Linux,
and edit startup configuration. It does not replace understanding
what the options do or choosing policies suited to your use case.
Runtime changes alone do not persist across remounts.
Additional reading:
* [Config/Options](../config/options.md)
+218
View File
@@ -0,0 +1,218 @@
# Downsides
## What are the downsides of using mergerfs?
mergerfs trades the features of an integrated storage system for the
flexibility of pooling independent filesystems. It is a filesystem
proxy, not RAID or a replacement for the underlying filesystems.
Some limitations are inherent to that design; others matter only for
particular workloads, configurations, or applications. Many can be
mitigated with appropriate settings or separate tools. The following
is a checklist of tradeoffs to evaluate, not a list of problems every
installation will experience.
### No built-in data protection or storage management
mergerfs does not provide redundancy, parity, integrity checks,
snapshots, or file versioning. If a drive fails, mergerfs cannot
recover the files on it. The other drives retain their independent
filesystems, but that failure isolation is not redundancy or a backup.
Protection must come from the underlying storage or separate tools,
such as [SnapRAID](https://www.snapraid.it) or
[NonRAID](https://github.com/qvr/nonraid).
It also does not automatically rebalance existing files when a branch
is added or space usage changes. If redistribution is needed, it must
be done separately.
See [Non-features](../index.md#non-features), [Project
Comparisons](../project_comparisons.md), and [How can I balance files
across the pool?](configuration_and_policies.md#how-can-i-balance-files-across-the-pool).
### No striping or contiguous pooled space
Each file lives entirely on one branch. Adding drives does not make a
single-file transfer faster, and a file must fit on one eligible
branch even if the pool reports enough total free space. For example,
two filesystems with 6GiB free each cannot hold one 10GiB file.
Concurrent access to different branches can still provide aggregate
throughput.
See [Project Comparisons](../project_comparisons.md#raid0-jbod-span-drive-concatenation-striping)
and [Performance](../performance.md#understanding-mergerfs-io-performance).
### Performance overhead
Filesystem requests generally pass through FUSE and the mergerfs
process before reaching the underlying filesystem. This adds latency
and CPU overhead compared to direct access. Directory listings and
other metadata operations can require work across multiple branches,
so their cost can grow as the pool grows. Page caching can also result
in data being cached both through mergerfs and on the underlying
filesystem.
[IO passthrough](../config/passthrough.md) can reduce data IO overhead,
but does not eliminate the work of combining branches. Databases, VM
images, and other latency-sensitive workloads should be benchmarked
with the intended configuration rather than assumed to perform like
direct access.
There is also memory overhead beyond page caching: request buffers,
temporary directory buffers, and FUSE node bookkeeping. Usage depends
on concurrency, buffer sizes, and the workload. Exporting the pool
over NFS requires retaining nodes with `never-forget-nodes=true`,
which can increase memory usage as more entries are accessed. This is
worth considering on memory-constrained systems or large NFS exports.
See [Resource Usage](../resource_usage.md) and [NFS exporting
mergerfs](../remote_filesystems.md#nfs-exporting-mergerfs).
See [Understanding mergerfs IO
Performance](../performance.md#understanding-mergerfs-io-performance),
[Benchmarking](../benchmarking.md), and [Why use
FUSE?](technical_behavior_and_limitations.md#why-use-fuse-why-not-a-kernel-based-solution).
### Placement and filesystem semantics need attention
A single mount point does not make the branches one filesystem.
[Policies](../config/functions_categories_policies.md) determine where
new files go and which copy is used when the same path exists on
multiple branches. Path-preserving policies and
[minfreespace](../config/minfreespace.md) can exclude branches from file
creation even while the pool has free space. mergerfs cannot know a
file's final size when it is created.
Hard links and renames still have underlying filesystem constraints.
Depending on the layout and policies, an operation can return `EXDEV`
(cross-device link) even within the pool. Applications which do not
handle that error may fail rather than fall back to copying.
Directory metadata is also selected by policy rather than necessarily
aggregated. With the default `getattr` policy of `ff`, a directory's
timestamp comes from the first matching branch. A media scanner which
uses that timestamp to skip unchanged directories can miss updates on
another branch. `func.getattr=newest` can help with timestamp-based
scans, but changes which branch supplies metadata; it does not change
which file other operations select.
Corresponding directories on different branches can have different
ownership or permissions. Since access uses the caller's credentials,
this can result in unexpected access errors or missing entries.
Keeping those permissions consistent and auditing with
`fsck.mergerfs` helps. Container UID/GID mappings and non-POSIX
branches may need additional attention; mergerfs does not bypass
their access restrictions.
See [Configuration and Policies](configuration_and_policies.md),
[rename and link](../config/rename_and_link.md), [Tips and
Notes](../tips_notes.md), and [How does mergerfs interact with user
namespaces?](compatibility_and_integration.md#how-does-mergerfs-interact-with-user-namespaces).
### Keeping drives spun down can be harder
Directory listings and file searches can query multiple branches,
potentially waking drives beyond the one containing the requested
file. If keeping disks asleep is important, this can increase power
usage, noise, and access latency. It is not a concern for SSDs or
drives which are kept spinning.
Caching and reducing background scans can limit some branch access,
but mergerfs alone cannot guarantee that unused disks stay asleep.
See [Limiting drive spinup](limit_drive_spinup.md) for the tradeoffs
and possible approaches.
### Direct branch changes and cached views
Using the underlying filesystems directly is supported, but changes
made outside the mergerfs mount are not forwarded as `inotify` or
`fanotify` events to applications watching the pool. Changes made
through the pool do generate notifications. If files are added
directly to branches, applications such as media servers may need
periodic scans, explicit refreshes, or watches on the branches
themselves.
Metadata caching introduces a separate visibility tradeoff: direct
branch changes may not appear through the pool until cached entries
expire. Longer cache durations can reduce filesystem queries at the
cost of less immediate visibility. Shorter durations or disabling the
relevant caches can help if direct branch changes are common. Avoid
uncoordinated writes to the same file through both paths, especially
when file caching is enabled.
See [Does inotify and fanotify
work?](compatibility_and_integration.md#does-inotify-and-fanotify-work),
[Can filesystems still be used
directly?](usage_and_functionality.md#can-filesystems-still-be-used-directly-outside-of-mergerfs-while-pooled),
and [Caching](../config/cache.md).
### Branch paths do not guarantee a particular disk is mounted
mergerfs works on paths and does not require each branch to be a
mount point. If an intended branch filesystem is not mounted but its
mount-point directory still exists, mergerfs can use that directory
on the filesystem beneath it. If writable and selected for creation,
new files can land there instead of on the intended disk, potentially
filling the root filesystem.
This is a deployment concern when pooling mounted filesystems, not a
problem with intentionally using ordinary directories as branches.
Configure mount dependencies or
`branches-mount-timeout` with `branches-mount-timeout-fail=true` if
startup should fail when branches are not ready. These are startup
safeguards, not continuous checks that the intended disks remain
mounted at runtime.
See [branches-mount-timeout](../config/branches-mount-timeout.md) and
[What happens if a filesystem disappears at
runtime?](usage_and_functionality.md#what-happens-if-a-filesystem-disappears-at-runtime).
### Dependence on branch responsiveness
mergerfs cannot make an unresponsive disk or remote filesystem
responsive. A request to a blocked branch blocks the thread handling
it. If enough threads are blocked, the pool can stop responding as
well. Independent on-disk filesystems do not guarantee that a failing
branch has no effect on access through the pool.
See [What happens if a branch filesystem
blocks?](technical_behavior_and_limitations.md#what-happens-if-a-branch-filesystem-blocks).
### Compatibility limitations
Some filesystem features are unavailable through FUSE or require
particular kernel versions and settings. For example, reflink requests
(`FICLONE` / `FICLONERANGE`) are not supported through the mergerfs
mount. Applications using `mmap` need page caching; older Linux or
mergerfs versions require it to be explicitly enabled.
Advisory locks taken through the pool are not forwarded to the
underlying branches. Local applications using only the mergerfs mount
still have kernel-managed locks, but those locks do not coordinate
with direct branch access or other hosts accessing a remote branch.
Inode identity also has tradeoffs when presenting multiple filesystems
as one. Most applications do not need special settings, but backup or
deduplication tools and NFS can be sensitive to inode values changing
when files move between branches or a policy selects a different
copy. [inodecalc](../config/inodecalc.md) offers different strategies:
for example, `path-hash` stabilizes identity by path for NFS, but
prevents applications from recognizing hard links by inode equality.
The hard links themselves still work.
On SELinux systems, per-file relabeling is unavailable through the
FUSE mount, so container bind-mount options `:z` and `:Z` do not
perform their usual relabeling. This does not mean containers cannot
use the pool: an appropriate SELinux policy or a mount-wide context
can allow access. A mount-wide context is not a substitute for
distinct per-file labels if the application requires them. See [Does
mergerfs work with SELinux
relabeling?](compatibility_and_integration.md#does-mergerfs-work-with-selinux-relabeling).
See [Technical Behavior and
Limitations](technical_behavior_and_limitations.md) and [Known Issues
and Bugs](../known_issues_bugs.md).
These tradeoffs do not require migrating data into a special format.
The underlying filesystems remain usable without mergerfs, and
mergerfs can be [removed without changing their
data](usage_and_functionality.md#can-mergerfs-be-removed-without-affecting-the-data).
+109 -34
View File
@@ -1,22 +1,31 @@
# Limiting drive spinup
## Does mergerfs do anything to prevent drive spinup?
No. In fact it makes it worse.
## How can I setup my system to limit drive spinup?
TL;DR: You really can't. Not through mergerfs alone. In fact mergerfs
makes an attempt to do so more complicated.
TL;DR: You really can't. Not through mergerfs alone. In fact using
mergerfs makes any general attempt to limit spinup more complicated.
mergerfs is a proxy. Despite having some caching behaviors it is not
designed to cache much more than metadata. It proxies calls between
client software and underlying filesystems. If a client makes a
request such as `open`, `readdir`, `stat`, etc. it must translate that
into something that makes sense across multiple filesystems. For
`readdir` that means running the call against all branches and
aggregating the output. For `open` that means finding the file to open
and doing so. The only way to find the file to open is to scan across
all branches and sort the results and pick one. There is no practical
way to do otherwise. Especially given so many mergerfs users expect
out-of-band changes to "just work."
mergerfs is a proxy. Despite having [some caching
behaviors](../config/cache.md) it is not designed to cache much more
than metadata. It proxies calls between client software and underlying
filesystems. If a client makes a request such as `open`, `readdir`,
`stat`, etc. it must translate that into something that makes sense
across multiple filesystems. For `readdir` that means running the same
call against all branches and aggregating the output. For `open` that
means finding the file to open and doing so. The only way to find the
file to open is to scan across all branches and sort the results and
pick one. There is some variability in how much searching is done due
to
[policies](../config/functions_categories_policies.md#policy-descriptions)
but not much. There is no practical way to do otherwise. Especially
given so many mergerfs users expect out-of-band changes to "just
work."
The best way to limit spinup of drives is to limit their usage at the
client level. Meaning keeping software from interacting with the
@@ -33,21 +42,24 @@ filesystems and on startup everything scanned and stored. From then on
it would have to carefully update all the same data the filesystems
do. It couldn't be kept in RAM because it would take up too much space
so it'd have to be on a SSD or other storage device. If anything
changed out of band it would break things in weird ways. It could
rescan on occasion but that would require spinning up
everything. Filesystem watches could be used to get updates when the
filesystem changes but that would allow for race conditions and
might keep the drives from spinning down. Something as "simple" as
keeping the current available free space on each filesystem isn't as
easy as one might think given reflinks, snapshots, and other block
level dedup technologies as well as the space used includes not just
raw file usage.
changed out of band it would break things in weird ways (and MANY
users depend on out of band changes working.) It could rescan on
occasion but that would require spinning up everything. Filesystem
watches could be used but since mergerfs is just another piece of
software interacting with the filesystem ALL mergerfs requests would
be echoed back to it causing lots of overhead throwing away a super
majority of events it triggered. And something as "simple" as keeping
the current available free space on each filesystem is not as easy as
one might think given reflinks, snapshots, and other block level dedup
technologies distort the meaning of usage values.
Even if all metadata (including xattrs) is cached some software will
open files (media like videos and audio) to check their
metadata. Granted a Plex or Jellyfin scan which may do that is
different from a random directory listing but is still something to
consider. Those "deep" scans can't be kept from waking drives.
Only if ALL metadata and all file locations were cached could mergerfs
then do very targeted file operations so as not to trigger a scan
across the pool. But that will break out of band interactions. It
would require accepting possibly bad behavior by space based policies
since space would be imprecisely computed. It would require long scans
of the filesystems on startup and a separate location to store all
this data.
### What if you only query already active drives?
@@ -56,7 +68,8 @@ Let's assume that is plausible (it isn't because some drives actually
will spin up if you ask if they are spun down... yes... really) you
would have to either cache all the metadata on the filesystem or treat
it like the filesystem doesn't exist. The former has all the problems
mentioned prior and the latter would break a lot of things.
mentioned prior and the latter would break a lot of things. Suddenly
whole filesystems worth of data would be missing because it spun down.
### Is there anything that can be done where mergerfs is involved?
@@ -65,15 +78,77 @@ Yes, but whether it works for you depends on your tolerance for the
complexity.
1. Cleanly separate writing, storing, and consuming the data.
1. Use a SSD or dedicated and limited pool of drives for downloads / torrents.
2. When downloaded move the files to the primary storage pool.
3. When setting up software like Plex, Jellyfin, etc. point to the
1. Use a SSD or dedicated and limited pool of drives for downloads / torrents.
2. When downloaded move the files to the primary storage pool.
3. When setting up software like Plex, Jellyfin, etc. point to the
underlying filesystems. Not mergerfs.
2. Add a bunch of bcache, lvmcache, dm-cache, or similar block level
cache to your setup. After a bit of use, assuming sufficient
storage space, you can limit the likelihood of the underlying
spinning disks from needing to be hit.
3. Tweak your system's settings around caching:
* mount all your filesystems with `noatime` to disable updating
"last accessed" time on files.
* mount all your filesystems with `nodiratime` to disable updating
"last accessed" time on directories.
* `sudo sysctl -w vm.vfs_cache_pressure=10`: Prioritizes keeping
file metadata cached in RAM.
* `sudo sysctl -w vm.dirty_ratio=40`: Allows up to 40% of system
memory to be filled with unwritten "dirty" data before forcing a
disk flush.
* `sudo sysctl -w vm.dirty_background_ratio=5`: Sets a lower
threshold (5%) for the kernel to start writing data in the
background without blocking applications.
* `sudo sysctl -w vm.dirty_writeback_centisecs=6000`: Increases
the interval between kernel writeback checks to 60 seconds
(default is usually 5).
* `sudo sysctl -w vm.dirty_expire_centisecs=12000`: Allows data to
stay "dirty" in RAM for up to 120 seconds before it must be
written to disk.
* To make `sysctl` values permanent add the above to
`/etc/sysctl.conf`.
* **Note:** for systems with limited RAM the above `sysctl` values
could cause issues.
* **Note:** if you increase the time for which data is stored in
RAM before being flushed to disk there is also an increased risk
that a sudden powerloss could lead to lost data.
4. Enable all the appropriate caches in mergerfs:
* [https://trapexit.github.io/mergerfs/latest/config/cache/](https://trapexit.github.io/mergerfs/latest/config/cache/)
* `cache.entry=86400`: cache file existence for 24h.
* `cache.negative-entry=86400`: cache file non-existence for 24h.
* `cache.attr=86400`: cache file metadata for 24h.
* `cache.symlinks=true`: enable symlink caching.
* `cache.readdir=true`: enable directory listing caching.
Remember too that while it may be a tradeoff you are willing to live
with there is decent evidence that spinning down drives puts increased
wear on them and can lead to their death earlier than otherwise.
### References
* [Documentation for /proc/sys/vm/](https://docs.kernel.org/admin-guide/sysctl/vm.html)
* [How to Configure vm.swappiness and Other Virtual Memory Parameters on RHEL](https://oneuptime.com/blog/post/2026-03-04-configure-vm-swappiness-virtual-memory-rhel-9/view)
### Tradeoffs
While it may be a trade off you are willing to live with there is
decent evidence that spinning down drives puts material increased wear
on them and can lead to their death earlier than otherwise. You may
end up saving money on electricity but spending more on having to
purchase new drives sooner.
## Alternatives
[Project Comparisons](../project_comparisons.md) lists a few projects
that are similar to mergerfs but offer caching. Please note that these
projects are very young and not general purpose like mergerfs. Their
compliance to POSIX filesystem rules is probably low enough that
certain behaviors may not work or worse lead to data corruption.
I'm not suggesting not to look at those projects but filesystem
development is complicated even when it appears to be just a layer on
top of another. Just be careful and do your research. That stands for
anything touching your data... mergerfs included.
To the degree that mergerfs can add features which help in this
situation, so long as it doesn't interfere with other fundamental
features, they may be added over time.
@@ -4,7 +4,7 @@
Yes. [links](https://en.wikipedia.org/wiki/Hard_link) (otherwise known
as hard links) are fundamentally supported by mergerfs. There is no
option to enable or disable links. All comments elswhere claiming they
option to enable or disable links. All comments elsewhere claiming they
do not work are related to the setup of the user's system.
See the [Intro to Filesystems](../intro_to_filesystems.md) page for more
+34 -1
View File
@@ -30,13 +30,19 @@ filesystem function is requested. Which file to select or act on.
See how [policies](../config/functions_categories_policies.md) work.
## Is adding or removing mergerfs to/from my setup destructive?
No. [See
below.](#can-mergerfs-be-used-with-filesystems-which-already-have-data)
## Can mergerfs be used with filesystems which already have data?
Yes. mergerfs is really just a proxy and does **NOT** interfere with
the normal form or function of the filesystems, mounts, or paths it
manages. It literally is interacting with your filesystems as any
other application does. It can not do anything that any other random
piece of software can't do.
piece of software can't do. There is no migration needed.
mergerfs is **not** a traditional filesystem that takes control over
the underlying drive, block device, or partition. mergerfs is **not**
@@ -44,6 +50,9 @@ RAID. It does **not** manipulate the data that passes through it. It
does **not** shard data across filesystems. It only shards some
**behavior** and aggregates others.
Think of it as a filesystem [reverse
proxy](https://en.wikipedia.org/wiki/Reverse_proxy).
## Can mergerfs be removed without affecting the data?
@@ -132,3 +141,27 @@ cache expires. mergerfs is not actively watching all branches for
changes and the kernel will have no way to know anything changed so as
to clear or ignore the cache. This is the same issue you can have with
[remote filesystems](../remote_filesystems.md).
## What are the performance characteristics of using mergerfs?
mergerfs does not stripe or shard data. Instead it acts like a reverse
proxy / multiplexer. As such it is not possible to have RAID like
performance improvements. However, its performance is not limited like
[block level concatenation / JBOD /
SPAN.](../project_comparisons.md#raid0-jbod-span-drive-concatenation-striping)
* Single file / single stream: Max throughput limited to speed of the
chosen underlying branch + FUSE overhead (usually 5–30% slower than
direct access; near native with [IO
passthrough](../config/passthrough.md)).
* Aggregate / multi-stream: Can approach sum of all branches' speeds
when many independent IO operations run in parallel (multiple files,
processes, or clients).
* Other factors: Latency-sensitive workloads suffer more (slow
branches or network drives hurt responsiveness); directory listing
and metadata ops slower due to combining branches.
Real-world performance varies heavily with workload, kernel, options,
and hardware. See [benchmarking](../benchmarking.md) and
[performance](../performance.md) section for testing tips.
+2 -2
View File
@@ -81,7 +81,7 @@ where it was created.
$ sudo touch /mnt/mergerfs/new-file
$ sudo getfattr -n user.mergerfs.allpaths /mnt/mergerfs/new-file
# file: mnt/mergerfs/new-file
user.mergerfs.allpaths="/mnt/hdd/drive1/new-file"
user.mergerfs.allpaths="/mnt/hdd/disk0/new-file"
```
If the location of the file is where it should be according to the
@@ -119,7 +119,7 @@ may be able to read some files and directories on one filesystem but
not another resulting in an incomplete set. And if one of the branches
it can access is empty then it will return an empty list.
Try using [mergerfs.fsck](https://github.com/trapexit/mergerfs-tools)
Try using [fsck.mergerfs](../tooling.md#fsckmergerfs)
tool to check for and fix inconsistencies in permissions. If you
aren't seeing anything at all be sure that the basic permissions are
correct. The user and group values are correct and that directories
+7 -5
View File
@@ -25,13 +25,13 @@ the overhead of traditional storage aggregation technologies.
**Key advantages:**
* Mix and match filesystems of any size,
* [Non-destructive](faq/usage_and_functionality.md#is-adding-or-removing-mergerfs-tofrom-my-setup-destructive) adding and removing of filesystems (even at runtime)
* Mix and match filesystems of any [size](faq/technical_behavior_and_limitations.md#are-there-any-size-limitations-to-files-directories-etc),
[type](faq/compatibility_and_integration.md#what-filesystems-can-be-used-as-branches),
or [underlying
device](faq/compatibility_and_integration.md#what-types-of-storage-devices-does-mergerfs-work-with)
* No parity calculations or rebuild times
* Does not require hard drives to be spinning if not in use
* Add or remove filesystems on the fly
* [Direct access to files](faq/usage_and_functionality.md#can-filesystems-still-be-used-directly-outside-of-mergerfs-while-pooled) on individual filesystems when needed
* Flexible [policies](config/functions_categories_policies.md) for controlling where new files are created
@@ -44,7 +44,8 @@ for more details.](project_comparisons.md)
## Features
* Logically combine numerous filesystems/paths into a single
mount point (JBOFS: Just a Bunch of FileSystems)
mount point (JBOFS: Just a Bunch of FileSystems, think filesystem
[reverse proxy](https://en.wikipedia.org/wiki/Reverse_proxy))
* Combine paths of the same or different filesystems
* Ability to add or remove filesystems/paths without impacting the
rest of the data
@@ -68,6 +69,7 @@ for more details.](project_comparisons.md)
* RAID like redundancy (see [SnapRAID](https://www.snapraid.it) and
[NonRAID](https://github.com/qvr/nonraid))
* RAID like performance improvements
* LVM/RAID style block device aggregation
* Data integrity checks, snapshots, file versioning
* Read/write overlays on top of read-only filesystems (like OverlayFS)
@@ -103,7 +105,7 @@ The way in which mergerfs behaves is controlled by the
```
A + B = C
/disk1 /disk2 /merged
/mnt/hdd/disk0 /mnt/hdd/disk1 /media
| | |
+-- /dir1 +-- /dir1 +-- /dir1
| | | | | |
@@ -140,7 +142,7 @@ Head to the [quick start guide](quickstart.md).
in one of the supported forums](support.md#contact-issue-submission)
and the docs will be updated.
* The search feature of MkDocs is not great. Searching for "literal
strings" will generally not work. Alernative solutions are being
strings" will generally not work. Alternative solutions are being
investigated.
* While not intended for end users nor completely accurate some might
find the AI generated docs at https://deepwiki.com/trapexit/mergerfs
+46 -9
View File
@@ -2,6 +2,30 @@
## mergerfs
### Advisory file locking not currently supported
mergerfs does not implement POSIX or BSD advisory file locks as
provided by FUSE. Doing so is complex because, as a union filesystem,
it would need to forward lock requests to the underlying branches,
which could result in blocking calls. Locally this is not a major
concern—the kernel still manages locks for applications that all go
through the mergerfs mount. However, if an application locks a file
via the mergerfs mount while another locks the same file directly on
the underlying filesystem, the locks will not interact as one might
expect. That usage pattern is uncommon, so it is generally not a
problem.
The issue becomes more significant if you rely on locks over a remote
filesystem like NFS across multiple hosts, because mergerfs never
actually makes a lock call to the underlying branch. While network
locks are possible, they are generally discouraged, so this may not be
a huge loss either.
That said, there is a strategy that could add lock support if there is
sufficient demand, though it would probably be opt-in for performance
and resource-usage reasons.
### FreeBSD version
* [https://wiki.freebsd.org/FUSEFS](https://wiki.freebsd.org/FUSEFS)
@@ -27,17 +51,30 @@
* ...
#### Host vs Container identity
### Host vs Container identity
While not a bug some users have found when using containers that
While **not** a bug some users have found when using containers that
supplemental groups defined inside the container don't work as
expected. Since mergerfs lives outside the container it is querying
the host's group database. Containers have their own user and group
definitions unless setup otherwise just as different systems would.
expected. Since mergerfs (typically) lives outside the container it is
querying the host's supplemental group database. Containers have their
own user and group definitions unless setup otherwise just as
different systems would.
Users should mount in the host group file into the containers or use a
standard shared user and groups technology like NIS or LDAP.
If using Docker or Podman containers with mergerfs running outside
then you can pass in `/etc/passwd` and `/etc/group`.
```
docker run \
-v /etc/passwd:/etc/passwd:ro \
-v /etc/group:/etc/group:ro \
-u $(id -u):$(id -g) \
-v /data:/data \
containerimage
```
### directory mtime is not being updated
@@ -55,15 +92,15 @@ If you always want the directory information from the one with the
most recent mtime then use the `newest` policy for `getattr`.
### 'mv /mnt/pool/foo /mnt/disk1/foo' removes 'foo'
### 'mv /mnt/pool/foo /mnt/hdd/disk0/foo' removes 'foo'
This is not a bug.
Run in verbose mode to better understand what's happening:
```
$ mv -v /mnt/pool/foo /mnt/disk1/foo
copied '/mnt/pool/foo' -> '/mnt/disk1/foo'
$ mv -v /mnt/pool/foo /mnt/hdd/disk0/foo
copied '/mnt/pool/foo' -> '/mnt/hdd/disk0/foo'
removed '/mnt/pool/foo'
$ ls /mnt/pool/foo
ls: cannot access '/mnt/pool/foo': No such file or directory
@@ -170,7 +207,7 @@ may need to disable the use of the inode in the software if possible
or change the [inodecalc](config/inodecalc.md) setting if possible.
Generally speaking, in modern setups where there are many non-fully
complient POSIX filesystem in use, reliance on inode to have a
compliant POSIX filesystems in use, reliance on inode to have a
specific meaning is not always reasonable. FUSE, for instance, has no
requirement that inode values need to be unique or that two files that
happen to be the same underlying file have the same inode.
+3
View File
@@ -29,6 +29,7 @@
* 2024-02-07 - [Designing & Deploying MANS - A Hybrid NAS Approach with SnapRAID, MergerFS, and OpenZFS](https://blog.muffn.io/posts/part-3-mini-100tb-nas)
* 2024-03-11 - [Using MergerFS to combine multiple hard drives into one unified media storage](https://fullmetalbrackets.com/blog/two-drives-mergerfs/)
* 2024-12-20 - [Pooling multiple drives on my Raspberry Pi with mergerfs](https://sebi.io/posts/2024-12-20-pooling-multiple-drives-with-mergerfs/)
* 2026-01-03 - [NAS-2026 | MergerFS and SnapRAID](https://github.com/DoTheEvo/NAS-MergerFS-SnapRAID)
## Videos
@@ -62,10 +63,12 @@
* 2024-01-05 - [OpenMediaVault MergerFS Tutorial (Portuguese)](https://www.youtube.com/watch?v=V6Yw86dRUPQ)
* 2024-02-19 - [Setup and Install MergerFS and SnapRAID (Part 1)](https://noted.lol/mergerfs-and-snapraid-setup-1/)
* 2024-02-22 - [Setup and Install MergerFS and SnapRAID (Part 2)](https://noted.lol/mergerfs-and-snapraid-setup-part-2/)
* 2024-05-29 - [Building a 26TB Offsite Backup Server! ft. Tailscale](https://www.youtube.com/watch?v=W_l82GF00UY)
* 2024-11-15 - [Meu servidor NAS - Parte 18: Recuperando um HD, recuperando o MergerFS e os próximos passos do NAS!](https://www.youtube.com/watch?v=5fy98kPzE3s)
* 2025-04-23 - [How to build the Perfect Media Server | Part 1 - The Tech Stack | mergerfs, SnapRAID, and docker.](https://www.youtube.com/watch?v=Yt67zz9p0FU)
* 2025-06-12 - [DIY Unraid-like functionality](https://www.youtube.com/watch?v=fR0XhaWvR1E)
* 2025-09-22 - [استفاده از Snapraid و MergerFS | جایگزین RAID5](https://www.youtube.com/watch?v=GA6OvaUBKXs)
* 2026-02-10 - [Never run out of space again! Using MergerFS with Immich to expand your storage seamlessly](https://www.youtube.com/watch?v=Vbu4WzWUikQ)
## Podcasts
+185
View File
@@ -0,0 +1,185 @@
# Migration
## Unraid
mergerfs and a parity-protected Unraid array can both present
directories from multiple filesystems as a single view, but they treat
storage membership very differently. With mergerfs, membership is a
property of its **configuration**: branches are paths and can be added
or removed at runtime without modifying the underlying filesystems.
With Unraid's main array, membership is part of the **array state**:
adding a data disk to an array with active parity normally requires
clearing and formatting it, and removing one normally requires parity
to be rebuilt.
Those array-level requirements, not the User Share filesystem itself,
are what shape migrations in either direction. When moving to mergerfs,
existing data filesystems can usually be reused as-is. When moving to
Unraid, populated filesystems can normally only be brought into the
array before parity is activated. See [Project
Comparisons](project_comparisons.md#unraid) for the comparison itself.
| Operation | mergerfs | Unraid array with parity |
| --- | --- | --- |
| Add an empty filesystem | Add its mount path as a branch | Clear if necessary, then format and add |
| Add a populated filesystem | Add its mount path; data is immediately visible | Not normally possible once parity is active without destroying its contents |
| Mix filesystem types | Any filesystem the host can mount | Supported array filesystem types only |
| Mix device sizes | Arbitrary | Data disks cannot exceed the parity disk size |
| Change membership while mounted | Supported through the [runtime interface](runtime_interface.md) | Array configuration normally requires stopping and restarting the array |
| Remove a filesystem and retain it unchanged | Remove the branch | Possible, but normal array removal requires parity to be rebuilt |
| Remove a filesystem without recalculating parity | Not applicable | Requires zeroing the disk, destroying its filesystem |
### Migrating from Unraid to mergerfs
Unraid array data disks each hold a complete, individually formatted
filesystem. `shfs` only unions their top-level directories, so there
is no striped or parity-dependent layout to unwind. After cleanly
stopping the array, unencrypted data filesystems can be mounted under
Linux and their mount points used directly as mergerfs branches. The
files do not need to be rewritten merely because the pooling
implementation changed.
For [encrypted Unraid disks](https://docs.unraid.net/unraid-os/system-administration/secure-your-server/securing-your-data/),
securely retain the existing passphrase or key file before retiring
Unraid. Open each disk's existing LUKS container with `cryptsetup`
using those credentials, then mount the inner filesystem through its
`/dev/mapper/` device and use that mount point as a branch. Do not
reformat the disk or create a new LUKS container. Without a working
passphrase or key file, the encrypted data cannot be accessed.
```
Unraid:
disk1 ─┐
disk2 ─┼─> shfs ─> /mnt/user
disk3 ─┘
+
parity
mergerfs:
disk1 ─┐
disk2 ─┼─> mergerfs ─> /mnt/storage
disk3 ─┘
```
Unraid's convention of representing shares as identically named
top-level directories also maps naturally to mergerfs' union
semantics.
* The parity disk is not a data filesystem and cannot become a branch
as-is. It can be reformatted as another data disk if parity
protection is no longer desired, or a separate parity solution such
as [SnapRAID](https://www.snapraid.it) or
[NonRAID](https://github.com/qvr/nonraid) can be used alongside
mergerfs if redundancy is still wanted.
* If migrating incrementally while keeping the remaining Unraid array
operational, the normal Unraid disk-removal and parity-rebuild
requirements still apply to each disk removed.
* If the entire array is being retired at once, those intermediate
Unraid parity rebuilds are unnecessary. Stop the array cleanly and
use the existing data filesystems as branches.
* Named pools need care. A pool backed by a single filesystem can be
mounted and used as one branch. A multi-device Btrfs or ZFS pool is
not equivalent to a collection of independent Unraid array disks: it
can either remain intact and be mounted as one mergerfs branch, or
its data must be migrated elsewhere before its component devices can
be repurposed as independent filesystems.
### Migrating from mergerfs to Unraid
mergerfs places no special metadata on its branches, so their
underlying filesystems remain unchanged. Direct reuse as Unraid array
data disks requires local disks with a supported filesystem **and a
partition layout accepted by the installed Unraid version**. A
supported filesystem type alone is not sufficient.
In particular, mergerfs can use a filesystem formatted directly on a
whole device, such as `/dev/sda`, without a partition table. Unraid
array data disks require a filesystem within a compatible partition.
Such unpartitioned filesystems cannot be imported directly: back up
their data, let Unraid partition and format the disks, then restore
the files. Creating a partition table over the existing filesystem is
not a safe conversion. Check Unraid's
[partitioning guidance](https://docs.unraid.net/unraid-os/troubleshooting/faq/)
before planning a no-copy migration.
Current versions of Unraid support importing populated disks when
constructing a new array, or after resetting the array configuration,
provided that the populated data disks are added **before parity is
activated**. Such a migration can therefore often be performed without
copying the data:
1. Stop mergerfs and cleanly unmount the branch filesystems.
2. Create or reset the Unraid array configuration.
3. Assign all populated mergerfs disks as data disks while no parity
disk is active.
4. Verify that the existing filesystems and data are recognized
correctly. If a populated disk is reported as unmountable, stop and
investigate; do not format it, as formatting destroys its contents.
5. Assign a parity disk at least as large as the largest data disk.
6. Build parity over the completed array.
Once parity has been established, that convenient path is effectively
closed. Adding another populated mergerfs disk to the parity-protected
array would require Unraid to clear the disk, destroying the existing
filesystem. A staged migration in which populated mergerfs disks are
added one at a time therefore generally requires either copying their
contents into the existing array first or resetting/reconstructing the
Unraid configuration so that all populated data disks are present
before parity is established.
Additional array-level constraints to keep in mind:
* Only filesystem types supported by Unraid can be assigned as array
data disks.
* No array data disk may be larger than the parity disk.
* Array configuration changes normally require stopping and restarting
the array. Removing a data disk normally requires a parity rebuild,
and Unraid's parity-preserving removal procedure works by erasing
the filesystem and zeroing the disk, so it is unsuitable when the
goal is to retain that filesystem for use elsewhere.
When migrating into individual **Unraid array data slots**, the
following configurations require copying data to suitable array
filesystems rather than reusing the source storage directly:
* Remote filesystems or filesystem types Unraid does not support.
* Storage that cannot be presented as a single array device, such as
multi-device Btrfs or ZFS pools or `mdadm` arrays.
This restriction does not rule out retaining storage as a named pool.
A compatible multi-device ZFS pool can be
[imported into Unraid](https://docs.unraid.net/unraid-os/advanced-configurations/optimize-storage/zfs-storage/#importing-zfs-pools-created-on-other-systems)
intact, without copying its data. Stop mergerfs and cleanly export the
pool on the source system. With the Unraid array stopped, use **Add
Pool**, assign all of the pool's devices (including support vdevs),
set **File System** to **Auto**, and start the array to import it.
If it is not recognized, investigate rather than format its devices.
The imported pool remains separate from the parity-protected array;
its redundancy comes from its ZFS layout, not Unraid array parity.
A branch pointing at a subdirectory does **not** by itself require a
data copy: eligibility depends on its underlying disk and filesystem.
For example, importing compatible disks that contributed
`/mnt/disk1/media` and `/mnt/disk2/media` preserves their contents, and
Unraid combines the top-level `media` directories into a
[User Share](https://docs.unraid.net/unraid-os/using-unraid-to/manage-storage/shares/#user-shares).
Review the desired share layout; deeper branch directories may need
to be reorganized within their existing filesystems rather than
copied to new storage.
### Summary
mergerfs treats storage membership as a property of its
**configuration**, while a parity-protected Unraid array treats it as
part of the **array state**. Moving an existing filesystem into or out
of mergerfs normally does not alter that filesystem at all. Moving
populated filesystems into or out of an active Unraid parity array can
require clearing disks, rebuilding parity, moving data, or
reconstructing the array configuration.
For semi-static data, mergerfs + SnapRAID provides pooling and parity
while retaining the ability to add and remove independently formatted
filesystems without modifying them. NonRAID provides an alternative
real-time parity layer which can likewise be combined with mergerfs.
+59
View File
@@ -48,6 +48,65 @@ before changing them to understand how functionality will change.
* use LVM and LVM cache to place a SSD in front of your HDDs
## Understanding mergerfs IO Performance
A common source of confusion when evaluating **mergerfs**' performance
is comparing it to traditional RAID or filesystem level pooling
technologies such as mdadm, ZFS, btrfs, or striped LVM. Those
solutions aggregate devices into a single virtual device and spread IO
across all members. Under favorable conditions that means the
performance of the pool is the sum of the performance of its members:
striped reads and writes hit multiple drives at once and the aggregate
throughput scales with the number of devices, up to the limits of the
interconnects (PCIe, SATA, network, etc.) which every member must
share.
**mergerfs** does not aggregate or stripe data. Every file lives
wholly on one branch and **mergerfs** simply routes access to it. As a
result **mergerfs** does not aggregate the performance of its branches
in the way RAID does. What it provides instead is **independent
performance across branches**.
* **Best case:** workloads which access different branches
concurrently perform independently of one another. If you have 3
filesystems on 3 drives and each is being actively used, the total
throughput can be A + B + C: the sum of each branch performing at
full speed. No single drive is a bottleneck for the others.
* **Worst case:** a single stream of IO - one process reading or
writing one file - performs at the speed of the single branch the
file is on, no differently than a lone filesystem. Additional
branches do not make a single file faster.
This means that when reasoning about expected performance the key
question is whether your workload is **single stream** or **multi
stream**:
* Single large file transfers, video playback, or one process doing
sequential IO will never exceed the speed of one branch regardless
of how many are mounted. If a single stream must be faster, put a
faster device (tiered cache, SSD, LVM cache) in front of the
branches or use an actual striped RAID layer beneath **mergerfs**'
branches.
* Concurrency across branches is where **mergerfs** shines. Many
parallel downloads, seeding torrents, multiple rsync streams,
multiple applications reading different files: these naturally
spread across branches and their throughput adds up.
There are practical upsides to this non-aggregated design as well:
* Drives of different sizes and speeds don't reduce the whole pool to
the slowest member. Each branch performs at its own speed.
* The failure of one drive only affects the files on that branch
rather than the integrity of the entire pool.
Finally, keep in mind that **mergerfs** is a proxy above the branches,
so measured throughput is the lesser of the branch's native performance
and any overhead introduced by **mergerfs** and FUSE. Use
[nullrw](config/options.md) or a ram disk as a branch to measure that
overhead separately, and see the [benchmarking
section](benchmarking.md) for how to benchmark meaningfully.
## Additional Reading
* [Benchmarking](benchmarking.md)
+497 -169
View File
@@ -2,24 +2,39 @@
There are no solutions, only trade-offs.
This page includes both direct pooling alternatives and adjacent storage
technologies that solve different problems. Compare their operating layer,
failure domain, redundancy model, and platform requirements rather than only
their apparent pool capacity.
## mhddfs
* [https://romanrm.net/mhddfs](https://romanrm.net/mhddfs)
mhddfs is, like mergerfs, a FUSE based filesystem which can pool
mhddfs is, like mergerfs, a FUSE-based filesystem which can pool
multiple filesystems together.
mhddfs had not been updated in over a decade and has known stability
and security issues. mergerfs provides a super set of mhddfs' features
and offers better performance. As of 2020 the author of mhddfs has
and security issues. mergerfs provides a superset of mhddfs' features
and offers better performance. As of 2020, the author of mhddfs has
[moved to using mergerfs.](https://romanrm.net/mhddfs#update)
Below is an example of mhddfs and mergerfs setup to work similarly.
* **Better than mergerfs for:** Nothing?
* **Worse than mergerfs for:** most modern deployments due to maintenance
status, feature coverage, and reliability.
`mhddfs -o mlimit=4G,allow_other /mnt/drive1,/mnt/drive2 /mnt/pool`
Below is an example of mhddfs and mergerfs set up to behave similarly.
mhddfs falls back to the drive with the most free space when every drive
has less than `mlimit` free; mergerfs with
[minfreespace](config/minfreespace.md) filters those branches instead and
may fail the create if none qualify.
`mergerfs -o minfreespace=4G,category.create=ff /mnt/drive1:/mnt/drive2 /mnt/pool`
```sh
mhddfs -o mlimit=4G,allow_other /mnt/hdd/disk0,/mnt/hdd/disk1 /mnt/pool
mergerfs -o minfreespace=4G,category.create=ff /mnt/hdd/disk0:/mnt/hdd/disk1 /mnt/pool
```
## aufs
@@ -27,47 +42,74 @@ Below is an example of mhddfs and mergerfs setup to work similarly.
* [https://aufs.sourceforge.net](https://aufs.sourceforge.net)
* [https://en.wikipedia.org/wiki/Aufs](https://en.wikipedia.org/wiki/Aufs)
aufs, another union filesystem, is a kernel based overlay filesystem
aufs, another union filesystem, is a kernel-based overlay filesystem
with basic file creation placement policies.
While aufs still is maintained it failed to be included in the
mainline kernel and is no longer available in most Linux distros
making it harder to get installed for the average user.
Compared with mergerfs, aufs focuses on kernel-level overlay and
copy-on-write behavior, while mergerfs focuses on userspace pooling of
existing filesystems with richer branch placement policies.
aufs failed to be included in the mainline kernel and is no longer
available in most Linux distros, making it harder to get installed for
the average user. Development has been largely dormant for years.
While aufs can often offer better peak performance due to being
primarily kernel based (at least when `passthrough.io` is disabled),
mergerfs provides more configurability and is generally easier to
use. mergerfs however does not offer the overlay / copy-on-write (CoW)
primarily kernel-based (at least when
[passthrough.io](config/passthrough.md) is disabled in mergerfs),
mergerfs provides more configurability and is generally easier to use.
mergerfs, however, does not offer the overlay / copy-on-write (CoW)
features which aufs has.
* **Better than mergerfs for:** kernel-level overlay/CoW workloads where
that behavior is required.
* **Worse than mergerfs for:** users needing a widely packaged,
actively-maintained, easy-to-deploy pooling solution.
## unionfs
* [https://unionfs.filesystems.org](https://unionfs.filesystems.org)
unionfs for Linux is a "stackable unification file system" which
functions like many other union filesystems. unionfs is no longer
maintained and was last released for Linux v3.14 back in 2014.
functions like many other union filesystems. It was last released for
Linux v3.14 back in 2014.
Compared with mergerfs, unionfs is an older kernel union filesystem
implementation while mergerfs is a modern, actively-maintained
FUSE-based pooling filesystem with broader policy controls.
Documentation is sparse so a comparison of features is not possible
but given the lack of maintenance and support for modern kernels there
is little reason to consider it as a solution.
* **Better than mergerfs for:** historical or legacy environments that
already depend on this specific kernel unionfs behavior.
* **Worse than mergerfs for:** practical modern deployments due to
limited maintenance, aging kernel support, and sparse documentation.
## unionfs-fuse
* [https://github.com/rpodgorny/unionfs-fuse](https://github.com/rpodgorny/unionfs-fuse)
unionfs-fuse is more like unionfs, aufs, and overlayfs than mergerfs
in that it offers overlay / copy-on-write (CoW) features. If you're
just looking to create a union of filesystems and want flexibility in
file/directory placement then mergerfs offers that whereas
unionfs-fuse is more for overlaying read/write filesystems over
read-only ones.
unionfs-fuse is a FUSE-based union filesystem focused on overlay
behavior, including copy-on-write (CoW) style layering of writable
content over read-only content.
Since unionfs-fuse, as the name suggests, is a FUSE based technology
it can be used without elevated privileges that kernel solutions such
as unionfs, aufs, and overlayfs require.
Compared with mergerfs, unionfs-fuse's primary feature is
overlay/CoW-style layering, while mergerfs' primary feature is
policy-based pooling and placement across multiple regular writable
filesystems.
As a FUSE filesystem, unionfs-fuse can often be mounted by an
unprivileged user when the system's FUSE configuration permits it. Kernel
solutions such as unionfs, aufs, and overlayfs normally require additional
privileges.
* **Better than mergerfs for:** overlay-style unions with copy-on-write
semantics in userspace.
* **Worse than mergerfs for:** flexible create-policy-based pooling of
many writable filesystems.
## overlayfs
@@ -76,12 +118,35 @@ as unionfs, aufs, and overlayfs require.
overlayfs is effectively the successor to unionfs, unionfs-fuse, and
aufs and is widely used by Linux container platforms such as Docker and
Podman. It was developed and is maintained by the same developer who
created FUSE.
Podman. Both overlayfs and FUSE were originally created by Miklos
Szeredi.
If your use case is layering a writable filesystem on top of read-only
filesystems then you should look first to overlayfs. Its feature set
however is very different from mergerfs and solve different problems.
however is very different from mergerfs and solves different problems.
* **Better than mergerfs for:** container and image-layering workflows
where copy-on-write semantics are central.
* **Worse than mergerfs for:** heterogeneous pooling use cases that need
flexible branch selection policies across many existing filesystems.
## fuse-overlayfs
* [https://github.com/containers/fuse-overlayfs](https://github.com/containers/fuse-overlayfs)
fuse-overlayfs implements overlayfs-style lower, upper, and work directories
in userspace. Its primary use case is rootless containers, where a kernel
overlayfs mount may not be available.
Like unionfs-fuse and kernel overlayfs, it provides layering and copy-up
semantics rather than mergerfs-style placement across a pool of writable
filesystems.
* **Better than mergerfs for:** rootless container and other unprivileged
overlay workloads.
* **Worse than mergerfs for:** general storage pooling across independent
writable filesystems.
## RAID0, JBOD, SPAN, drive concatenation, striping
@@ -91,32 +156,31 @@ however is very different from mergerfs and solve different problems.
* [SPAN](https://en.wikipedia.org/wiki/Non-RAID_drive_architectures#Concatenation_(SPAN,_BIG))
* [striping](https://en.wikipedia.org/wiki/Data_striping)
These are block device technologies which in some form aggregate
devices into what appears to be a singular device on which a
traditional filesystem can be used. The filesystem has no
understanding of the underlying block layout and should one of those
underlying devices fail or be removed the filesystem will be missing
that chunk which could contain critical information and the whole
filesystem may become unrecoverable. Even if the data from the
filesystem is recoverable it will take using specialized tooling to do
so.
"JBOD" is ambiguous: it can also mean disks exposed independently with
no aggregation. In this section it refers only to concatenated or spanned
layouts.
In contrast, with mergerfs you can format devices as one normally
would or take existing filesystems and then combine them in a pool to
These are block device technologies which, in some form, aggregate
devices into what appears to be a single device on which a traditional
filesystem can be used. The filesystem has no understanding of the
underlying block layout and, should one of those underlying devices
fail or be removed, the filesystem will be missing that chunk, which
could contain critical information, and the whole filesystem may
become unrecoverable. Even if the data from the filesystem is
recoverable, doing so may require specialized tooling.
In contrast, with mergerfs you can format devices as you normally
would, or take existing filesystems and combine them in a pool to
aggregate their storage. The failure of any one device will have no
impact on the other devices. The downside to mergerfs' technique is
the fact you don't actually have contiguous space as large as if you
used those other technologies. Meaning you can't create a file greater
than 1TB on a pool of two 1TB filesystems.
impact on the other devices. The downside to mergerfs' approach is
that you do not actually have contiguous space as large as you would
with those other technologies. That means you cannot create a file
larger than 1TB on a pool of two 1TB filesystems.
## BTRFS Single Data Profile
[BTRFS'](https://btrfs.readthedocs.io) `single` data profile is
similar to RAID0 but BTRFS can more explicitly place data and metadata
on multiple devices. However, like RAID0 and similar technologies if a
single device fails you may lose all data in the pool. BTRFS does have
some abilities to recover data but this is not guaranteed.
* **Better than mergerfs for:** creating one contiguous block device and
maximizing single-file size/performance characteristics.
* **Worse than mergerfs for:** failure isolation and straightforward
per-disk recovery when a device dies.
## RAID5, RAID6
@@ -125,36 +189,120 @@ some abilities to recover data but this is not guaranteed.
* [RAID6](https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6)
RAID5, RAID6, and similar technologies aggregate multiple devices at
the block level and offer 1 or more live parity calculations that can
allow the pool to continue to run should 1 device fail and rebuild the
data once the device is replaced.
the block level and provide one or more live parity schemes that can
allow the pool to continue to run should one device fail and rebuild
the data once the device is replaced.
mergerfs offers no parity or redundancy features so in that regard the
technologies are not comparable. [SnapRAID](https://www.snapraid.it)
or [nonraid](https://github.com/qvr/nonraid) can be used in
combination with mergerfs to provide redundancy. Unlike traditional
RAID5 or RAID6 SnapRAID works with drives of different sizes and can
have more than 2 parity drives. However, parity calculations are not
done in real-time. However, nonraid is realtime.
technologies are not comparable. [SnapRAID](https://www.snapraid.it) or
NonRAID (see [below](#nonraid)) can be used in combination with mergerfs
to provide redundancy. Unlike traditional RAID5 or RAID6, SnapRAID works
with drives of different sizes and can have more than 2 parity drives.
Parity calculations with SnapRAID are not done in real-time but
NonRAID's are.
For more details and comparison of SnapRAID to related technologies see [https://www.snapraid.it/compare](https://www.snapraid.it/compare).
For more details and comparison of SnapRAID to related technologies
see [https://www.snapraid.it/compare](https://www.snapraid.it/compare).
* **Better than mergerfs for:** live parity protection and continued array
operation through configured disk failures.
* **Worse than mergerfs for:** incremental mixed-size pooling with simple
per-filesystem independence.
## UnRAID
## LVM
* [https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)](https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux))
* [https://sourceware.org/lvm2](https://sourceware.org/lvm2)
LVM is a block-level volume management system for Linux. It combines
physical devices into volume groups and creates logical volumes on top,
with features like snapshots, resizing, and thin provisioning.
Compared with mergerfs, LVM operates below the filesystem while mergerfs
operates above filesystems. With LVM, filesystems are created on logical
volumes and do not know about the underlying disks. If a linear or
striped logical volume spans multiple disks and one disk fails, the
filesystem can become unreadable. With mergerfs, each disk typically has
its own independent filesystem, so a single disk failure affects only the
data on that disk.
* **Better than mergerfs for:** block-level features like thin
provisioning, snapshots, and online resizing.
* **Worse than mergerfs for:** incremental mixed-disk pooling where
per-disk failure isolation and per-disk native access are priorities.
## mdadm
* [https://en.wikipedia.org/wiki/Mdadm](https://en.wikipedia.org/wiki/Mdadm)
* [https://docs.kernel.org/admin-guide/md.html](https://docs.kernel.org/admin-guide/md.html)
`mdadm` is Linux userspace tooling for configuring and managing software
RAID arrays provided by the kernel `md` driver. It can build RAID0/1/4/5/6/10
and related layouts, monitor arrays, and handle rebuild workflows.
Like RAID technologies discussed above, `mdadm` arrays aggregate devices
at the block layer and present one virtual device to a filesystem. That
can provide performance and/or redundancy depending on RAID level, but it
also means the filesystem depends on the array layout. mergerfs does not
provide block-level RAID; it pools already-formatted filesystems and can
be combined with tools like SnapRAID or NonRAID when redundancy is needed.
* **Better than mergerfs for:** kernel software RAID with real-time
redundancy and array-level performance tuning.
* **Worse than mergerfs for:** flexible mixed-size drive pooling where
per-filesystem independence and simple incremental expansion matter.
## device mapper
* [https://en.wikipedia.org/wiki/Device_mapper](https://en.wikipedia.org/wiki/Device_mapper)
* [https://docs.kernel.org/admin-guide/device-mapper/index.html](https://docs.kernel.org/admin-guide/device-mapper/index.html)
Device mapper is a kernel framework for creating virtual block devices
from one or more underlying devices. Technologies such as LVM,
`dm-crypt`, `dm-cache`, `dm-thin`, and multipath are built on top of it.
Because device mapper works at the block layer, it is not directly
comparable to mergerfs. It is complementary: you can use device mapper
for block-level functionality (encryption, caching, thin provisioning,
etc.) and then place filesystems on those devices, which mergerfs can
later pool if desired.
* **Better than mergerfs for:** block-layer transformations such as
transparent encryption, caching, and provisioning features.
* **Worse than mergerfs for:** presenting one unified pathname-level view
across many independent, already-formatted filesystems.
## Unraid
* [https://unraid.net](https://unraid.net)
UnRAID is a full OS and offers a (FUSE based?) filesystem which
provides a union of filesystems like mergerfs but with the addition of
live parity calculation and storage. Outside parity calculations
mergerfs offers more features and due to the lack of real-time parity
calculation can have higher peak performance. Some users also prefer
an open source solution.
Unraid is a full OS platform that offers pooled storage with integrated
real-time parity. Compared with mergerfs, its main advantage is that it
bundles pooling and parity in one product, while mergerfs is a
standalone Linux pooling filesystem without built-in redundancy. For
some users mergerfs' Linux distro-agnostic and fully open-source
approach is also preferable.
For semi-static data mergerfs + [SnapRAID](http://www.snapraid.it)
provides a similar, but not real-time, solution. NonRAID (see below)
is a fork of UnRAID's parity calculation solution and can also be used
with mergerfs.
For semi-static data, mergerfs + [SnapRAID](https://www.snapraid.it)
provides a similar, but not real-time, solution.
[NonRAID](https://github.com/qvr/nonraid) (see [below](#nonraid)) is a
fork of Unraid's parity calculation solution and can also be used with
mergerfs.
Moving between the two is not symmetric because mergerfs membership is
a configuration change while Unraid array membership is part of the
array state. See [Migration](migration.md) for details.
* **Better than mergerfs for:** turnkey pooling plus integrated real-time
parity in one product.
* **Worse than mergerfs for:** non-destructive incremental addition and
removal of populated filesystems, arbitrary branch types and filesystem
layouts, and users who prefer a Linux distro-agnostic, fully open-source
component approach.
## NonRAID
@@ -162,170 +310,350 @@ with mergerfs.
* [https://github.com/qvr/nonraid](https://github.com/qvr/nonraid)
[NonRAID](https://github.com/qvr/nonraid#how-it-works) is a fork of
UnRAID's storage array kernel driver. Effectively it is like RAID5 at
Unraid's storage array kernel driver. Effectively it is like RAID5 at
the block device layer without treating the collection of devices as a
single device. Each device gets a virtual block device associated with
it which can be formatted with the filesystem of the user's choice and
will be individually accessible. Similar to SnapRAID but real-time.
Compared with mergerfs, NonRAID adds kernel-level real-time parity while
still allowing per-disk filesystems, whereas mergerfs focuses on pooling
views and placement policies without parity.
Given each device has its own filesystem like a traditional setup it can
be used with mergerfs for those looking for a unified view.
* **Better than mergerfs for:** real-time parity protection without
traditional RAID striping behavior.
* **Worse than mergerfs for:** users who want the simplest setup and do
not need parity, since it adds operational complexity.
## ZFS
* [https://en.wikipedia.org/wiki/ZFS](https://en.wikipedia.org/wiki/ZFS)
* [https://openzfs.github.io/openzfs-docs](https://openzfs.github.io/openzfs-docs)
ZFS is an advanced filesystem with many features including RAID5/6
like redundency.
ZFS is an advanced filesystem that combines storage pooling,
snapshots, data integrity checking, and built-in RAID-like redundancy
all in one package. It offers features like transparent compression,
encryption, and automatic data repair.
mergerfs is very different from ZFS. mergerfs is intended to provide
flexible pooling of arbitrary filesystems (local or remote), of
arbitrary sizes, and arbitrary filesystems. Particularly in `write
once, read many` use cases such as bulk media storage. Where data
integrity and backup is managed in other ways. In those use cases ZFS
can introduce a number of costs and limitations as described
[here](http://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html),
[here](https://markmcb.com/2020/01/07/five-years-of-btrfs/), and
[here](https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSWhyNoRealReshaping).
Compared with mergerfs, ZFS is a full storage stack that manages disks
directly with integrated integrity and redundancy features, while
mergerfs is a lightweight FUSE layer that pools existing filesystems
without built-in redundancy.
That said given mergerfs' flexibility it is common to see it used in
combination with ZFS where critical data is written to a RAIDZ pool
and bulk media stored on traditional filesystems pooled via mergerfs.
* **Better than mergerfs for:** integrated data integrity, snapshots,
checksumming, and built-in redundancy.
* **Worse than mergerfs for:** low-overhead incremental pooling of
existing filesystems with minimal planning.
## ZFS AnyRAID
[ZFS
AnyRAID](https://hexos.com/blog/introducing-zfs-anyraid-sponsored-by-eshtek)
is a feature being developed for ZFS which is intended to provide more
flexibility in ZFS pools. Allowing a mix of capacity disks to have
greater capacity than traditional RAID and would allow for partial
upgrades while keeping live redundancy.
* [Announcement](https://docs.hexos.com/blog/2025-05-22)
* [Development update](https://docs.hexos.com/blog/2026-7-15)
This ZFS feature, as of mid-2025, is extremely early in its development
and there are no timelines or estimates for when it may be released.
ZFS AnyRAID is under development to support mixed-capacity disks while
retaining ZFS redundancy. The planned layouts include mirrored and RAID-Z
variants, along with rebalancing and pool contraction work.
Availability is still landing in OpenZFS. Published notes indicate initial
mirror layouts may ship in a 2.4.x release, with broader RAID-Z variants,
rebalancing, and pool contraction targeted for later releases. Treat
release and downstream product-integration dates as roadmap targets
rather than commitments.
Where available, AnyRAID would offer some of the mixed-size flexibility
associated with mergerfs while retaining redundancy inside ZFS. mergerfs
provides mixed-size, add-drives-anytime pooling today, but without
built-in redundancy; SnapRAID or NonRAID can provide separate
protection.
* **Better than mergerfs for:** users who want mixed-capacity flexibility
and built-in live redundancy in one filesystem stack.
* **Worse than mergerfs for:** users who need production availability now or
want to pool existing filesystems without migrating data.
## bcachefs
* [https://bcachefs.org](https://bcachefs.org)
bcachefs is a modern copy-on-write filesystem for Linux designed for
multi-device storage aggregation. It includes integrated features such as
checksumming, snapshots, compression, encryption, and redundancy.
Compared with mergerfs, bcachefs is a full integrated filesystem, while
mergerfs is a lightweight pooling layer over existing filesystems. Beginning
with Linux 6.18, bcachefs is no longer distributed in the mainline kernel; the
project now distributes it as an external DKMS module. It remains actively
developed, but deployments must account for kernel compatibility and its
shorter operational history.
* **Better than mergerfs for:** integrated CoW filesystem features,
checksumming, snapshots, compression, and built-in multi-device
redundancy.
* **Worse than mergerfs for:** pooling existing filesystems without
reformatting, mainline-kernel availability, and per-disk native access.
## Btrfs multi-device filesystems
* [https://btrfs.readthedocs.io/en/latest/Volume-management.html](https://btrfs.readthedocs.io/en/latest/Volume-management.html)
Btrfs can combine multiple devices in one filesystem and select separate
allocation profiles for data and metadata. The `single` data profile spreads
one copy of data across available devices without redundancy; other profiles
provide mirroring, striping, or parity with different capacity and failure
trade-offs.
Compared with mergerfs, multi-device Btrfs is one integrated filesystem,
while mergerfs pools separate existing filesystems. Losing a device from a
`single` profile can make the whole Btrfs filesystem incomplete. Redundant
profiles change that risk, but the member devices still are not independently
mountable filesystems.
* **Better than mergerfs for:** integrated checksumming, snapshots, balancing,
and selectable redundancy profiles.
* **Worse than mergerfs for:** pooling existing mixed filesystem types,
per-disk native access, and failure isolation without redundancy.
## StableBit's DrivePool
* [https://stablebit.com](https://stablebit.com)
* [https://stablebit.com/DrivePool/Features](https://stablebit.com/DrivePool/Features)
DrivePool is a commercial filesystem pooling technology for
Windows. If you are looking to use Windows then DrivePool is a good
option. Functionally the two projects work a bit
differently. DrivePool always writes to the filesystem with the most
free space and later rebalances. mergerfs does not currently offer
rebalance but chooses a branch at file/directory create
time. DrivePool's rebalancing can be done differently in any directory
and has file pattern matching to further customize the
behavior. mergerfs, not having rebalancing does not have these
features, but similar features are planned for mergerfs v3. DrivePool
has builtin file duplication which mergerfs does not natively support
(but can be done via an external program.)
DrivePool is a commercial storage pooling product for Windows that
combines multiple drives into a single volume and includes balancing
and optional duplication capabilities.
There are a lot of misc differences between the two projects but most
features in DrivePool can be replicated with external tools in
combination with mergerfs.
Compared with mergerfs, DrivePool emphasizes integrated balancing and
duplication features in a Windows-first product, while mergerfs
emphasizes Linux FUSE pooling with policy-based create placement and
external-tool composability.
* **Better than mergerfs for:** Windows users needing built-in
rebalancing and native file duplication features.
* **Worse than mergerfs for:** Linux-native deployments and users who
prefer open-source tooling and composable external utilities.
## Plan9 binds
## Windows Storage Spaces
* [https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/overview](https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/overview)
Windows Storage Spaces is Microsoft's block-level storage virtualization
platform for Windows. It aggregates physical disks into storage pools from
which virtual disks can be created with simple, mirror, or parity layouts.
Supported Windows Server configurations can also use faster media as a
Storage Bus Cache.
Unlike mergerfs, Storage Spaces manages the member devices below the
filesystem and stripes or mirrors data according to the selected layout.
This can provide integrated resilience and one contiguous volume, but member
disks are not independently readable filesystems. mergerfs instead pools
existing filesystems and leaves their files directly accessible.
* **Better than mergerfs for:** Windows environments needing integrated
pooling, resilience, and tiering in the platform stack.
* **Worse than mergerfs for:** Linux hosts and lightweight pooling where
minimal overhead and per-filesystem independence are preferred.
## Plan 9 binds
* [https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs#Union_directories_and_namespaces](https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs#Union_directories_and_namespaces)
Plan9 has the native ability to bind multiple paths/filesystems
together to create a setup similar to simplified union
filesystem. Such bind mounts choose files in a "first found" manner
based on the order they are configured similar to mergerfs' `ff`
policy. Similarly, when creating a file it will be created on the
first directory in the union.
Plan 9 has a native bind mechanism that can compose paths and
filesystems into a single namespace view, including union-like
directory behavior.
Compared with mergerfs, Plan 9 binds are an OS-native namespace feature
with simple ordered behavior, while mergerfs provides Linux-focused
filesystem pooling with a larger set of configurable branch policies.
Plan 9 isn't a widely used OS so this comparison is mostly academic.
* **Better than mergerfs for:** native Plan 9 environments where
namespace composition is a core OS feature.
* **Worse than mergerfs for:** Linux users due to ecosystem relevance and
practical deployment availability.
## 9P
* [https://en.wikipedia.org/wiki/9P_(protocol)](https://en.wikipedia.org/wiki/9P_(protocol))
[9P](https://en.wikipedia.org/wiki/9P_(protocol)) is a filesystem
protocol from the Plan 9 operating system used to access files over a
network between systems.
Compared with mergerfs, 9P is a network filesystem protocol, while
mergerfs is a local Linux FUSE filesystem for pooling directories and
filesystems under one mount with branch policies. Like the Plan 9 binds
comparison above, this is mostly academic for typical mergerfs users.
* **Better than mergerfs for:** lightweight network filesystem protocol
use between compatible systems.
* **Worse than mergerfs for:** local Linux filesystem pooling because it
does not provide mergerfs-like branch policies or pooling semantics.
## SnapRAID pooling
* [https://www.snapraid.it/manual](https://www.snapraid.it/manual)
[SnapRAID](https://www.snapraid.it/manual) has a pooling feature which
creates "a read-only virtual view of all the files in your array using
symbolic links."
As mentioned in the description this "view" is just the creation of
Compared with mergerfs, SnapRAID pooling is a symlink-generated,
read-only view refreshed by SnapRAID operations, while mergerfs is a
live mounted filesystem that supports normal read/write behavior.
As mentioned in the description, this "view" is just the creation of
the same directory layout with symlinks to all files. This means that
reads (and writes) to files are at native speeds but limited in that
it can not practically be used as a target for writing new files and
is only updated when `snapraid pool` is run. Note that some software
treat symlinks differently than regular files. For instance some
backup software will skip symlinks by default.
reads (and writes) to files are at native speeds, but it is not
practical to use as a target for writing new files, and it is only
updated when `snapraid pool` is run. Note that some software treats
symlinks differently than regular files. For instance, some backup
software will skip symlinks by default.
mergerfs has the feature [symlinkify](config/symlinkify.md) which
provides a similar behavior but is more flexible in that it is not
read-only. That said there can still be some software that won't like
that kind of setup.
* **Better than mergerfs for:** read-only, native-speed directory views
of SnapRAID-managed content.
* **Worse than mergerfs for:** general read/write pooled filesystem use
and always-live view updates.
## rclone union
* [https://rclone.org/union](https://rclone.org/union)
rclone's [union](https://rclone.org/union) backend allows you to
create a union of multiple rclone backends and was inspired by
[mergerfs](https://rclone.org/union/#behavior-policies). Given rclone
knows more about the underlying backend than mergerfs could it can be
more efficient than creating a similar union with `rclone mount` and
mergerfs.
[mergerfs](https://rclone.org/union/#behavior-policies). Because
rclone knows more about the underlying backends than mergerfs can, it
can be more efficient than creating a similar union with `rclone mount`
and mergerfs.
However, it is not uncommon to see users setup rclone mounts and
Compared with mergerfs, rclone union is backend-aware and optimized for
cloud/remote storage abstractions, while mergerfs is optimized for local
POSIX filesystem pooling and branch policies.
However, it is not uncommon to see users set up rclone mounts and
combine them with local or other remote filesystems using mergerfs
given the differing feature sets and focuses of the two projects.
* **Better than mergerfs for:** remote-backend-heavy workloads where
backend-aware behavior improves cloud storage handling.
* **Worse than mergerfs for:** local POSIX pooling scenarios where direct
filesystem semantics and broad local tool compatibility matter.
## distributed filesystems
* AFS
* Ceph/CephFS
* GlusterFS
* LizardFS
* MooseFS
* [AFS](https://www.openafs.org)
* [Ceph/CephFS](https://docs.ceph.com/en/latest/cephfs/)
* [GlusterFS](https://www.gluster.org)
* [LizardFS](https://lizardfs.com)
* [MooseFS](https://moosefs.com)
* etc.
Distributed remote filesystems come in many forms. Some offering POSIX
filesystem compliance and some not. Some providing remote block
Distributed remote filesystems come in many forms. Some offer POSIX
filesystem compliance and some do not. Some provide remote block
devices or object stores on which a POSIX or POSIX-like filesystem is
built on top of. Some which are effectively distributed union
filesystems with duplication.
built. Some are effectively distributed union filesystems with
duplication.
These filesystems almost always require a significant amount of
compute to run well and are typically deployed on their own
hardware. Often in an "orchestrators" + "workers" configuration across
numerous nodes. This limits their usefulness for casual and homelab
users. There could also be issues with network congestion and general
performance if using a single network and that network is slower than
the storage devices.
Compared with mergerfs, distributed filesystems focus on multi-node
storage, replication, and cluster behavior, while mergerfs focuses on
single-host local pooling of existing filesystems.
While possible to use a distributed filesystem to combine storage
devices (and typically to provide redundancy) it will require a more
complicated setup and more compute resources than mergerfs (while also
offering a different set of capabilities.)
* **Better than mergerfs for:** multi-node or geographically distributed
workloads with built-in replication and high availability.
* **Worse than mergerfs for:** single-host and homelab pooling where low
overhead and operational simplicity are key.
## 9P
## nofs
[9P, the Plan 9 Filesystem
Protocol,](https://en.wikipedia.org/wiki/9P_(protocol)) is a protocol
developed for the Plan 9 operating system to help expand on the Unix
idea that everything should be a file. The protocol made its way to
other systems and is still widely used. As such 9P is not directly
comparable to mergerfs but more so to FUSE which mergerfs uses. FUSE
is also a filesystem protocol (though designed for kernel <->
userspace communication rather than over a network). FUSE, even more
than the
[9P2000.L](https://github.com/chaos/diod/blob/master/protocol.md)
variant of 9P, is focused primarily on supporting Linux filesystem
features.
* [https://github.com/chapmanjacobd/nofs](https://github.com/chapmanjacobd/nofs)
mergerfs leverages FUSE but could have in theory leveraged 9P with a
reduction in features.
nofs is a tool that provides mergerfs-like functionality for combining
multiple filesystems/directories into a unified view but does so
entirely through subcommands which replicate traditional POSIX
commands rather than providing a proper filesystem. It takes a lot of
inspiration from mergerfs in that it supports policy-based branch
selection (mfs, ff, pfrd, epmfs, and others), read-only and no-create
branch modes, and recreates POSIX-like commands such as `ls`, `find`,
`which`, `cp`, `mv`, and `rm`.
While 9P has [extensive
usage](https://docs.kernel.org/filesystems/9p.html) in certain
situations its use in modern userland Linux systems is limited. FUSE
has largely replaced use cases that may have been implemented with 9P
servers in the past.
Compared with mergerfs, nofs is command-driven rather than a mounted
filesystem, so mergerfs is the option for transparent system-wide
pooling through normal POSIX filesystem calls.
Given its design nofs is not suited for general usage as third-party
applications will not be able to take advantage of the unioning
behavior it offers. It is primarily for simpler situations where
mergerfs cannot be used.
* **Better than mergerfs for:** constrained environments where a FUSE
mount is not possible but command-style union behavior is acceptable.
* **Worse than mergerfs for:** transparent system-wide filesystem pooling
usable by arbitrary third-party applications.
## PolicyFS
* [https://policyfs.org](https://policyfs.org)
* [https://github.com/hieutdo/policyfs](https://github.com/hieutdo/policyfs)
PolicyFS is a Linux FUSE storage daemon that unifies multiple storage
paths under a single mountpoint, similarly to mergerfs. Its
distinguishing features are explicit path-pattern-based routing rules
for placement, and an optional SQLite metadata index that can serve
`readdir` and `getattr` operations without spinning up HDDs.
Compared with mergerfs, PolicyFS emphasizes explicit path-based routing,
scheduled SSD-to-HDD movement, and optional metadata indexing. mergerfs
emphasizes a broader policy-driven pooling model with longer operational
history and wider application compatibility. mergerfs does not try to prevent
metadata operations from waking branches because doing so fundamentally
[conflicts with other features.](faq/limit_drive_spinup.md)
PolicyFS's indexed storage deliberately defers some delete, rename, and
attribute operations until a maintenance job runs. Out-of-band changes require
the index to be refreshed, and its documentation identifies limits around
`O_DIRECT`, advisory locks, NFS re-export, and full POSIX coverage. Those are
material trade-offs, not only maturity differences.
* **Better than mergerfs for:** explicit path-based routing, scheduled
tiering, and metadata indexing designed to reduce HDD spin-ups.
* **Worse than mergerfs for:** broad POSIX compatibility, immediate mutation
semantics on every branch, and long-established operational behavior.
## Greyhole
* [https://www.greyhole.net](https://www.greyhole.net)
* [https://github.com/gboudreau/Greyhole](https://github.com/gboudreau/Greyhole)
Greyhole is an open-source storage pooling application that works via
Samba. Rather than implementing a FUSE filesystem, it hooks into Samba
VFS to intercept file operations and distribute files across multiple
drives, with optional per-share redundancy.
Compared with mergerfs, Greyhole is Samba-centric and includes
integrated share-level redundancy, while mergerfs is a general-purpose
Linux FUSE mount for local pooling via standard POSIX filesystem calls.
* **Better than mergerfs for:** Samba-centric environments that need
built-in per-share redundancy in the same tool.
* **Worse than mergerfs for:** general-purpose local filesystem pooling
outside Samba, and for heavy small-file churn workloads.
+74 -15
View File
@@ -1,4 +1,4 @@
# QuickStart
# Quick Start
## Install
@@ -62,28 +62,84 @@ For some suggestions on branch setup see [the page on
branches](config/branches.md#branch-setup).
### Web UI
[mergerfs-webui](tooling.md#mergerfs-webui) provides a browser-based
alternative for creating and managing mounts on Linux. It can edit
startup configuration and inspect and change settings on running
mounts. See the
[mergerfs-webui documentation](https://github.com/trapexit/mergerfs-webui#readme)
for installation and secure access guidance. Runtime changes alone do
not persist across remounts.
### Command Line
```
# Two positional options: `:` colon separated branches and mountpoint
mergerfs -o opt[,opt...] /branch0:/branch1 /mountpoint
mergerfs -o opt[,opt...] /mnt/hdd/disk0:/mnt/hdd/disk1:/mnt/hdd/disk2 /media
# or list of positional options where the last is the mountpoint
mergerfs -o opt[,opt...] /branch0 /branch1 /mountpoint
mergerfs -o opt[,opt...] /mnt/hdd/disk0 /mnt/hdd/disk1 /mnt/hdd/disk2 /media
# or set mountpoint via -o mountpoint
mergerfs -o mountpoint=/mountpoint /branch0 /branch1
mergerfs -o mountpoint=/media /mnt/hdd/disk0 /mnt/hdd/disk1 /mnt/hdd/disk2
```
```
mergerfs -o cache.files=off,category.create=pfrd,func.getattr=newest,dropcacheonclose=false /mnt/hdd0:/mnt/hdd1 /media
mergerfs -o cache.files=off,category.create=pfrd,func.getattr=newest,dropcacheonclose=false /mnt/hdd/disk0:/mnt/hdd/disk1:/mnt/hdd/disk2 /media
```
### /etc/fstab
```
/mnt/hdd0:/mnt/hdd1 /media mergerfs cache.files=off,category.create=pfrd,func.getattr=newest,dropcacheonclose=false 0 0
/mnt/hdd/disk0:/mnt/hdd/disk1:/mnt/hdd/disk2 /media mergerfs cache.files=off,category.create=pfrd,func.getattr=newest,dropcacheonclose=false 0 0
```
### systemd mount dependencies
On a `systemd` based system, no custom service or setup script is
needed when the branch filesystems and mergerfs are all configured in
`/etc/fstab`. Add
[`x-systemd.requires-mounts-for=`](https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html#x-systemd.wants-mounts-for=)
once for each branch mount target in the mergerfs entry:
```fstab
LABEL=disk0 /mnt/hdd/disk0 ext4 defaults 0 2
LABEL=disk1 /mnt/hdd/disk1 ext4 defaults 0 2
LABEL=disk2 /mnt/hdd/disk2 ext4 defaults 0 2
/mnt/hdd/disk0:/mnt/hdd/disk1:/mnt/hdd/disk2 /media mergerfs cache.files=off,category.create=pfrd,func.getattr=newest,dropcacheonclose=false,x-systemd.requires-mounts-for=/mnt/hdd/disk0,x-systemd.requires-mounts-for=/mnt/hdd/disk1,x-systemd.requires-mounts-for=/mnt/hdd/disk2 0 0
```
Replace the example labels and filesystem types with those used by the
system. `systemd` creates `Requires=` and `After=` dependencies from
the mergerfs mount to `mnt-hdd-disk0.mount`,
`mnt-hdd-disk1.mount`, and `mnt-hdd-disk2.mount`. It starts those
mounts first and does not start the mergerfs mount if a required mount
fails. This checks the mount units, not merely whether the
`/mnt/hdd/diskN` directories exist.
After editing `/etc/fstab`, reload the generated units:
```shell
sudo systemctl daemon-reload
```
The dependencies apply the next time `/media` is mounted and on
subsequent boots.
When using a service such as the
[`mergerfs-media.service`](#systemd-simple) example below instead of an
`/etc/fstab` mergerfs entry, add the equivalent directive to its
`[Unit]` section:
```systemd
RequiresMountsFor=/mnt/hdd/disk0 /mnt/hdd/disk1 /mnt/hdd/disk2
```
`RequiresMountsFor=` adds both the requirement and ordering
dependencies, so separate `Requires=` and `After=` entries for these
mounts are unnecessary.
### /etc/fstab w/ config file
For more complex setups it can be useful to separate out the config.
@@ -114,10 +170,9 @@ resolve the symlinks and use the real path.
`ls -lh /etc/mergerfs/branches/media/*`
```text
lrwxrwxrwx 1 root root 21 Aug 4 2023 hdd00 -> /mnt/hdd/hdd00
lrwxrwxrwx 1 root root 21 Aug 4 2023 hdd01 -> /mnt/hdd/hdd01
lrwxrwxrwx 1 root root 21 Aug 4 2023 hdd02 -> /mnt/hdd/hdd02
lrwxrwxrwx 1 root root 21 Aug 4 2023 hdd03 -> /mnt/hdd/hdd03
lrwxrwxrwx 1 root root 14 Aug 4 2023 disk0 -> /mnt/hdd/disk0
lrwxrwxrwx 1 root root 14 Aug 4 2023 disk1 -> /mnt/hdd/disk1
lrwxrwxrwx 1 root root 14 Aug 4 2023 disk2 -> /mnt/hdd/disk2
```
### systemd (simple)
@@ -127,6 +182,7 @@ lrwxrwxrwx 1 root root 21 Aug 4 2023 hdd03 -> /mnt/hdd/hdd03
```systemd title="mergerfs-media.service" linenums="1"
[Unit]
Description=mergerfs /media service
RequiresMountsFor=/mnt/hdd/disk0 /mnt/hdd/disk1 /mnt/hdd/disk2
After=local-fs.target network.target
[Service]
@@ -138,8 +194,9 @@ ExecStart=/usr/bin/mergerfs \
-o category.create=pfrd \
-o func.getattr=newest \
-o dropcacheonclose=false \
/mnt/hdd0 \
/mnt/hdd1 \
/mnt/hdd/disk0 \
/mnt/hdd/disk1 \
/mnt/hdd/disk2 \
/media
ExecStop=/usr/bin/umount /media
# Or if you need fusermount
@@ -193,8 +250,9 @@ WantedBy=default.target
```systemd title="mergerfs-media.service" linenums="1"
[Unit]
Description=mergerfs /media service
RequiresMountsFor=/mnt/hdd/disk0 /mnt/hdd/disk1 /mnt/hdd/disk2
Requires=setup-for-mergerfs.service
After=local-fs.target network.target prepare-for-mergerfs.service
After=local-fs.target network.target setup-for-mergerfs.service
[Service]
Type=simple
@@ -205,8 +263,9 @@ ExecStart=/usr/bin/mergerfs \
-o category.create=pfrd \
-o func.getattr=newest \
-o dropcacheonclose=false \
/mnt/hdd0 \
/mnt/hdd1 \
/mnt/hdd/disk0 \
/mnt/hdd/disk1 \
/mnt/hdd/disk2 \
/media
ExecStop=/usr/bin/umount /media
# Or if you need fusermount
+28 -21
View File
@@ -2,9 +2,9 @@
## Projects using mergerfs
* [Lakka.tv](https://lakka.tv/): A turnkey software emulation Linux
* [Lakka.tv](https://lakka.tv): A turnkey software emulation Linux
distribution. Used to pool user and local storage. Also includes my
other project [Opera](https://retroarch.com/). A 3DO emulator.
other project [Opera](https://retroarch.com). A 3DO emulator.
* [OpenMediaVault](https://www.openmediavault.org): A network attached
storage (NAS) solution based on Debian Linux. They provide plugins
to manage mergerfs.
@@ -13,42 +13,47 @@
mergerfs to create pools from existing filesystems.
* [ZimaOS](https://github.com/IceWhaleTech/zimaos-rauc): A more
commercially focused NAS OS by the authors of CasaOS at [Ice
Whale](https://www.zimaboard.com/).
* [Cosmos Cloud](https://cosmos-cloud.io/): Cosmos "take the chore out
Whale](https://www.zimaboard.com).
* [Cosmos Cloud](https://cosmos-cloud.io): Cosmos "take the chore out
of selfhosting, with automated maintenance and fully secured setup
out of the box. It even integrates to your existing setup."
* [MOS](https://mos-official.net): "Lightweight, modular server OS for
NAS, Containers & VMs."
## Software and services commonly used with mergerfs
* [snapraid](https://www.snapraid.it/): a backup program designed for
* [mergerfs-webui](tooling.md#mergerfs-webui): a browser-based tool for
managing mergerfs mounts, runtime settings, and startup configuration
on Linux.
* [snapraid](https://www.snapraid.it): a backup program designed for
disk arrays, storing parity information for data recovery in the
event of up to six disk failures.
* [nonraid](https://github.com/qvr/nonraid): NonRAID is a fork of the
unRAID system's open-source md_unraid kernel driver enabling
UnRAID-style storage arrays with parity protection outside of the
commercial UnRAID system.
* [rclone](https://rclone.org/): a command-line program to manage
Unraid system's open-source md_unraid kernel driver enabling
Unraid-style storage arrays with parity protection outside of the
commercial Unraid system.
* [rclone](https://rclone.org): a command-line program to manage
files on cloud storage. It is a feature-rich alternative to cloud
vendors' web storage interfaces. rclone's
[union](https://rclone.org/union/) feature is based on mergerfs
[union](https://rclone.org/union) feature is based on mergerfs
policies.
* [ZFS](https://openzfs.org/): A popular filesystem and volume
* [ZFS](https://openzfs.org): A popular filesystem and volume
management platform originally part of Sun Solaris and later ported
to other operating systems. It is common to use ZFS with
mergerfs. ZFS for important data and mergerfs pool for replacable
media.
* [Proxmox](https://www.proxmox.com/): Proxmox is a popular, Debian
* [Proxmox](https://www.proxmox.com): Proxmox is a popular, Debian
based, virtualization platform. Users tend to install mergerfs on
the host and pass the mount into containers.
* [UnRAID](https://unraid.net): "Unraid is a powerful, easy-to-use
* [Unraid](https://unraid.net): "Unraid is a powerful, easy-to-use
operating system for self-hosted servers and network-attached
storage." While UnRAID has its own union filesystem it isn't
uncommon to see UnRAID users leverage mergerfs given the differences
storage." While Unraid has its own union filesystem it isn't
uncommon to see Unraid users leverage mergerfs given the differences
in the technologies. There is a [plugin available by
Rysz](https://forums.unraid.net/topic/144999-plugin-mergerfs-for-unraid-support-topic/)
Rysz](https://forums.unraid.net/topic/144999-plugin-mergerfs-for-unraid-support-topic)
to ease installation and setup.
* [TrueNAS SCALE](https://www.truenas.com/truenas-scale/): An
* [TrueNAS SCALE](https://www.truenas.com/truenas-scale): An
enterprise focused NAS operating system with OpenZFS support. A Some
users are requesting mergerfs be [made part
of](https://forums.truenas.com/t/add-unionfs-or-mergerfs-and-rdam-enhancement-then-beat-all-other-nas-systems/23218)
@@ -58,7 +63,7 @@
details [on their
wiki](https://cloudboxes.io/wiki/how-to/apps/set-up-mergerfs-using-ssh):
on how to setup mergerfs.
* [QNAP](https://www.myqnap.org/product/mergerfs-apache83/): A company
* [QNAP](https://www.myqnap.org/product/mergerfs-apache83): A company
known for their turnkey, consumer focused NAS devices. Someone has
created builds of mergerfs for different QNAP devices.
@@ -71,11 +76,13 @@ Linux](https://repology.org/project/mergerfs/versions) distributions
and FreeBSD.
Note: Any non-rolling release based distro is likely to have
out-of-date versions.
out-of-date versions. `deb` and `rpm` based distro users should
probably use the [mergerfs project's provided
packages.](https://github.com/trapexit/mergerfs/releases)
* [Debian](https://packages.debian.org/bullseye/mergerfs)
* [Debian](https://packages.debian.org/search?keywords=mergerfs&searchon=names&suite=all&section=all)
* [Ubuntu](https://launchpad.net/ubuntu/+source/mergerfs)
* [Fedora](https://rpmsphere.github.io/)
* [Fedora](https://rpmsphere.github.io)
* [T2](https://t2sde.org/packages/mergerfs)
* [Alpine](https://pkgs.alpinelinux.org/packages?name=mergerfs&branch=edge&repo=&arch=&maintainer=)
* [Gentoo](https://packages.gentoo.org/packages/sys-fs/mergerfs)
+18 -19
View File
@@ -55,9 +55,9 @@ NFS export settings:
other software. By setting `softerr` or `soft` the NFS client will
timeout eventually and return an error.
* `softreval`: NFS client will serve up cached data after `retrans`
attempts to revalidate the data. Helps with intermitent network
attempts to revalidate the data. Helps with intermittent network
issues.
* `timeo=150`: Timeout till retrying request.
* `timeo=150`: Timeout till retrying request.
* `retrans=3`: Number of retrying a request.
* `rsize=131072`: Read size.
* `wsize=131072`: Write size.
@@ -76,21 +76,20 @@ but if you run into problems it may be worth trying Samba/SMB.
**mergerfs settings:**
* `never-forget-nodes=true` or `remember-nodes=SECONDS`
* `never-forget-nodes=true`
* `inodecalc=path-hash`
* `lazy-umount-mountpoint=false`
`noforget` (or `remember-nodes` with a large value such as `86400` for
a full day) is needed because NFS uses the `name_to_handle_at` and
`open_by_handle_at` functions which allow a program to keep a
reference to a file without technically having it open in the typical
sense. The problem is that FUSE has no way to know that NFS has a
handle that it will later use to open the file again. As a result, it
is possible for the kernel to tell mergerfs to forget about the file
node and should NFS ever ask for that node's details in the future it
would have nothing to respond with. Keeping nodes around forever (or
however long is set with `remember-nodes`) is not ideal but at the
moment the only way to manage the situation.
`never-forget-nodes=true` is needed because NFS uses the
`name_to_handle_at` and `open_by_handle_at` functions which allow a
program to keep a reference to a file without technically having it
open in the typical sense. The problem is that FUSE has no way to know
that NFS has a handle that it will later use to open the file
again. As a result, it is possible for the kernel to tell mergerfs to
forget about the file node and should NFS ever ask for that node's
details in the future it would have nothing to respond with. Keeping
nodes around until they are unlinked, removed, or replaced is not ideal
but at the moment the only way to manage the situation.
`inodecalc=path-hash` is needed because NFS is sensitive to
out-of-band changes. FUSE doesn't care if a file's inode value changes
@@ -120,11 +119,11 @@ value. An easy way to generate a random value is to use the command
line tool `uuid` or `uuidgen` or through a website such as
[uuidgenerator.net](https://www.uuidgenerator.net/).
`no_root_squash` is required for the same reason mergerfs needs to run
as `root`. Certain behaviors of mergerfs require control over the
filesystem that only `root` can preform. If squashing is enabled, or
mergerfs was running as non-root, it would be unable to perform
certain function and you will receive permission errors.
`no_root_squash` is not *technically* required but suggested. If NFS
is root squashing then any request that an app running as root makes
will show up as a different user to mergerfs. This is perfectly fine
so long as the client software using NFS are not running as root and
expect to have root like privileges.
## SMB / CIFS
+4 -3
View File
@@ -12,7 +12,7 @@
depending on [fuse-msg-size](config/fuse-msg-size.md)
* buffers allocated temporarily for reading directories
* FUSE nodes
* noforget forgotten nodes
* never-forget-nodes=true retained nodes, until unlinked or removed
## Management
@@ -23,8 +23,9 @@
value to -990.
* mergerfs increases [its available file descriptor and file size
limit.](https://www.man7.org/linux/man-pages/man3/setrlimit.3p.html)
* mergerfs lowers its [scheduling
* mergerfs sets its [scheduling
priority](https://man7.org/linux/man-pages/man3/setpriority.3p.html)
to -10 ([by default](config/options.md))
(nice value) to -10 ([by default](config/options.md)), raising its
CPU scheduling priority above the default of 0.
* The [readahead](config/readahead.md) values of mergerfs itself and
managed filesystems can be modified.
+12 -7
View File
@@ -14,6 +14,11 @@ getting and setting config and issuing certain commands. The other are
files found on the filesystem for querying certain `mergerfs` specific
information about them.
For a browser-based interface to inspect and change runtime settings,
manage branches and policies, and issue commands, see
[mergerfs-webui](tooling.md#mergerfs-webui). Runtime changes still need
to be saved in startup configuration if they should persist.
## .mergerfs pseudo file
@@ -35,7 +40,7 @@ configure the mounting of mergerfs (/etc/fstab, systemd, etc.).
### Command Line Tooling
Extended attributes is prevelant enough that there are common tools
Extended attributes are prevalent enough that there are common tools
available for interacting with them.
In Debian / Ubuntu distributions you can get the tools
@@ -110,7 +115,7 @@ situation and continue to be able to access the xattr interface.
```
[trapexit:/mnt/mergerfs] $ getfattr -d .mergerfs
user.mergerfs.branches="/mnt/a=RW:/mnt/b=RW"
user.mergerfs.branches="/mnt/hdd/disk0=RW:/mnt/hdd/disk1=RW"
user.mergerfs.minfreespace="4294967295"
user.mergerfs.moveonenospc="false"
...
@@ -122,14 +127,14 @@ user.mergerfs.category.search="mfs"
[trapexit:/mnt/mergerfs] $ getfattr -n user.mergerfs.category.create .mergerfs
user.mergerfs.category.search="prfd"
[trapexit:/mnt/mergerfs] $ setfattr -n user.mergerfs.branches -v "'+</mnt/c=RO .mergerfs
[trapexit:/mnt/mergerfs] $ setfattr -n user.mergerfs.branches -v '+</mnt/hdd/disk2=RO' .mergerfs
[trapexit:/mnt/mergerfs] $ getfattr -n user.mergerfs.branches .mergerfs
user.mergerfs.branches="/mnt/c=RO:/mnt/a=RW:/mnt/b=RW"
user.mergerfs.branches="/mnt/hdd/disk2=RO:/mnt/hdd/disk0=RW:/mnt/hdd/disk1=RW"
```
### Commands
There are a number of commands / behaviors which can be triggerd by
There are a number of commands / behaviors which can be triggered by
writing ([setfattr](https://linux.die.net/man/1/setfattr),
[setxattr](https://linux.die.net/man/2/setxattr)) particular xattr
keys of `/mountpoint/.mergerfs`. These keys do not show up in key
@@ -172,7 +177,7 @@ to set them will result in an error.
```
[trapexit:/mnt/mergerfs] $ getfattr -n user.mergerfs.fullpath test
user.mergerfs.fullpath="/mnt/a/test"
user.mergerfs.fullpath="/mnt/hdd/disk0/test"
[trapexit:/mnt/mergerfs] $ getfattr -n user.mergerfs.allpaths .
user.mergerfs.allpaths="/mnt/a\000/mnt/b\000/mnt/c"
user.mergerfs.allpaths="/mnt/hdd/disk0\000/mnt/hdd/disk1\000/mnt/hdd/disk2"
```
+24 -2
View File
@@ -76,7 +76,7 @@ sudo rpm -i mergerfs-<ver>.fc<rel>.<arch>.rpm
## SUSE/OpenSUSE
Avaliable at [filesystems repo](https://build.opensuse.org/package/show/filesystems/mergerfs)
Available at [filesystems repo](https://build.opensuse.org/package/show/filesystems/mergerfs)
if you have not added the repo (e.g. in case of OpenSUSE Tumbleweed, rel=openSUSE_Tumbleweed):
@@ -135,11 +135,33 @@ docker pull ghcr.io/trapexit/mergerfs:TAG
docker run --device=/dev/fuse --cap-add=SYS_ADMIN -v /mnt/to-merge:/mnt/to-merge:rshared -v /mnt/mergerfs:/mnt/mergerfs:z,shared ghcr.io/trapexit/mergerfs:TAG -f '/mnt/to-merge/*' '/mnt/mergerfs'
```
or with docker compose
```
services:
mergerfs:
image: ghcr.io/trapexit/mergerfs:TAG
container_name: mergerfs
restart: unless-stopped
privileged: true
cap_add:
- SYS_ADMIN
devices:
- /dev/fuse:/dev/fuse
command: -f '/mnt/to-merge/*' '/mnt/mergerfs'
volumes:
- /mnt/to-merge:/mnt/to-merge:rshared
- /mnt/mergerfs:/mnt/mergerfs:z,shared
```
* `--device=/dev/fuse`: Pass in host FUSE device.
* `--cap-add=SYS_ADMIN`: Give proper permissions to mount.
* `-v /mnt/mergerfs:/mnt/mergerfs:z,shared`: `z` and `shared` will
allow the mount to be shared between the container and host and
therefore other containers.
therefore other containers. Note that once mergerfs is mounted
inside the container the resulting FUSE filesystem does not
support SELinux relabeling. See [Does mergerfs work with SELinux
relabeling?](../faq/compatibility_and_integration.md#does-mergerfs-work-with-selinux-relabeling)
* `-v /mnt/to-merge:/mnt/to-merge:rshared`: The mounts from the host
to merge within the container.
+22 -17
View File
@@ -24,32 +24,37 @@ directly.](mailto:support@spawn.link)**
* Solution already ruled out and why.
* The details from the output of the
[mergerfs.collect-info](tooling.md#mergerfscollect-info) tool.
* Alternatively:
* Version of mergerfs: `mergerfs --version`
* mergerfs settings / arguments: from fstab, systemd unit, command
line, OMV plugin, etc.
* Version of the OS: `uname -a` and `lsb_release -a`
* List of branches, their filesystem types, sizes (before and after
issue): `df -h`, `lsblk -o NAME,FSTYPE,FSSIZE,SIZE,MOUNTPOINTS,RM,RO,ROTA`
* **All** information about the relevant paths and files: permissions, ownership, etc.
* **All** information about the application making the requests: version, uid/gid
* **All** information about the relevant paths and files: permissions,
ownership, etc. Some of this is collected by the tool but only the
branch root details.
* **All** information about the application making the requests:
version, uid/gid. Try though to replicate any issues with common
core utilities.
* Try to recreate the problem in the simplest way using standard
programs: `ln`, `mv`, `cp`, `ls`, `dd`, etc. With the most simple
mergerfs setup possible such as using only a single branch, well
supported branch filesystem, maybe reduce to using a single thread,
etc.
* Runtime environment:
* Is mergerfs running within a container?
* Are the client apps using mergerfs running in a container?
* A `strace` of the app having problems:
* `strace -fvTtt -s 256 -o /tmp/app.strace.txt <cmd>`
* A `strace` of mergerfs **while** the program is trying to do
whatever it is failing to do:
* `strace -fvTtt -s 256 -p <mergerfsPID> -o /tmp/mergerfs.strace.txt`
* **Precise** directions on replicating the issue. Do not leave
**anything** out.
* Try to recreate the problem in the simplest way using standard
programs: `ln`, `mv`, `cp`, `ls`, `dd`, etc. With the most simple
mergerfs setup possible.
* A `strace` of mergerfs **before** and **while** the program is
trying to do whatever it is failing to do:
* `strace -fvTtt -s 256 -p <mergerfsPID> -o
/tmp/mergerfs.strace.txt`
* If mergerfs is blocking or appears stuck:
* `gdb -q --batch -ex "set pagination off" -ex "thread apply all
bt full" -ex quit --pid <mergerfsPID> > /tmp/mergerfs.stacktrace.txt`
* **Precise** step-by-step directions on replicating the issue. Do not
leave **anything** out.
### Contact / Issue submission
In order of preference.
* github.com: [https://github.com/trapexit/mergerfs/issues](https://github.com/trapexit/mergerfs/issues)
* discord: [https://discord.gg/MpAr69V](https://discord.gg/MpAr69V)
* reddit: [https://www.reddit.com/r/mergerfs](https://www.reddit.com/r/mergerfs)
+2 -1
View File
@@ -14,7 +14,8 @@
* If you do not see some directories and files you expect, policies
seem to skip branches, you get strange permission errors, etc. be
sure the underlying filesystems' permissions are all the same. Use
`mergerfs.fsck` to audit the filesystem for out of sync permissions.
[fsck.mergerfs](tooling.md#fsckmergerfs) to audit the filesystem for
out of sync permissions.
* If you still have permission issues be sure you are using POSIX
compliant filesystems. mergerfs doesn't generally make exceptions
for FAT, NTFS, or other non-POSIX filesystem.
+31 -9
View File
@@ -1,5 +1,28 @@
# Tooling
## mergerfs-webui
[mergerfs-webui](https://github.com/trapexit/mergerfs-webui) is a
browser-based tool for viewing and managing mergerfs mounts on Linux.
It is a separate project with a single executable containing both the
web server and the web page.
It can:
* Install mergerfs and create mounts through `/etc/fstab` or systemd
`.mount` units.
* Discover running mounts, mount and unmount them, and inspect and
change their runtime settings, branches, and policies.
* Edit startup configuration in `/etc/fstab`, systemd `.mount` units,
and referenced mergerfs ini files, with previews of structured
changes before saving.
* Issue runtime commands and set up the web UI as a systemd service.
[Runtime changes](runtime_interface.md) are not automatically
persistent; startup configuration must also be updated if changes
should survive a remount.
## mergerfs.collect-info
A tool included in recent releases of `mergerfs` which collects
@@ -75,7 +98,7 @@ the read/write workflow, such as
[moveonenospc](config/moveonenospc.md), will no longer work.
Also, understand that this will only work on dynamically linked
software (a that dynamically linked with the same general libc version
software (that is dynamically linked with the same general libc version
as the software being used with it.) Anything statically compiled will
not work. Many GoLang and Rust apps are statically compiled.
@@ -83,8 +106,8 @@ The library will not interfere with non-mergerfs filesystems. The
library is written to always fallback to returning the mergerfs opened
file on error.
While the library was written to account for a number of edgecases
there could be some yet accounted for so please report any oddities.
While the library was written to account for a number of edge cases
there could be some yet unaccounted for so please report any oddities.
Thank you to
[nohajc](https://github.com/nohajc/mergerfs-io-passthrough) for
@@ -105,14 +128,15 @@ to `.bashrc`.
### Docker and Podman usage
Assume `/mnt/fs0` and `/mnt/fs1` are pooled with mergerfs at `/media`.
Assume `/mnt/hdd/disk0` and `/mnt/hdd/disk1` are pooled with mergerfs
at `/media`.
All mergerfs branch paths _must_ be bind mounted into the container at
the same path as found on the host so the preload library can see
them.
**NOTE:** Since a container can have its own OS setup there is no
guarentee that `preload.so` from the host install will be compatible
guarantee that `preload.so` from the host install will be compatible
with the loader found in the container. If that is true it simply
won't work and shouldn't cause any issues.
@@ -134,8 +158,8 @@ docker run \
-e LD_PRELOAD=/usr/lib/mergerfs/preload.so \
-v /usr/lib/mergerfs/preload.so:/usr/lib/mergerfs/preload.so:ro \
-v /media:/media \
-v /mnt/fs0:/mnt/fs0 \
-v /mnt/fs1:/mnt/fs1 \
-v /mnt/hdd/disk0:/mnt/hdd/disk0 \
-v /mnt/hdd/disk1:/mnt/hdd/disk1 \
ubuntu:latest \
bash
```
@@ -160,8 +184,6 @@ Environment=LD_PRELOAD=/usr/lib/mergerfs/preload.so
behaviors that can be build on top of mergerfs. They may not
have all the features you are looking for.**
* mergerfs.ctl: A tool to make it easier to query and configure mergerfs at runtime
* mergerfs.fsck: Provides permissions and ownership auditing and
the ability to fix them (should use `fsck.mergerfs` instead)
* mergerfs.dedup: Will help identify and optionally remove duplicate files
* mergerfs.dup: Ensure there are at least N copies of a file across the pool
* mergerfs.balance: Rebalance files across filesystems by moving them from the most filled to the least filled
+6
View File
@@ -10,6 +10,12 @@ mike deploy --push "latest" "master" --ignore-remote-status
for tag in $(git tag -l | sort --version-sort --reverse)
do
case "${tag}" in
*-rc*)
continue
;;
esac
if git ls-tree -r --name-only "${tag}" | grep -q 'mkdocs.yml'; then
git checkout "${tag}"
mike deploy --push "${tag}" --ignore-remote-status
+19 -17
View File
@@ -63,8 +63,8 @@ nav:
- setup/installation.md
- setup/upgrade.md
- setup/build.md
- terminology.md
- intro_to_filesystems.md
- Terminology: terminology.md
- Intro to Filesystems: intro_to_filesystems.md
- Config:
- config/options.md
- config/deprecated_options.md
@@ -96,17 +96,18 @@ nav:
- config/export-support.md
- config/kernel-permissions-check.md
- config/lazy-umount-mountpoint.md
- error_handling_and_logging.md
- resource_usage.md
- runtime_interface.md
- remote_filesystems.md
- tips_notes.md
- known_issues_bugs.md
- project_comparisons.md
- benchmarking.md
- performance.md
- tooling.md
- extended_usage_patterns.md
- Error Handling and Logging: error_handling_and_logging.md
- Resource Usage: resource_usage.md
- Runtime Interface: runtime_interface.md
- Remote Filesystems: remote_filesystems.md
- Tips and Notes: tips_notes.md
- Known Issues and Bugs: known_issues_bugs.md
- Project Comparisons: project_comparisons.md
- Migration: migration.md
- Benchmarking: benchmarking.md
- Performance: performance.md
- Tooling: tooling.md
- Extended Usage Patterns: extended_usage_patterns.md
- FAQ:
- faq/why_isnt_it_working.md
- faq/configuration_and_policies.md
@@ -115,8 +116,9 @@ nav:
- faq/compatibility_and_integration.md
- faq/technical_behavior_and_limitations.md
- faq/have_you_considered.md
- faq/downsides.md
- faq/limit_drive_spinup.md
- related_projects.md
- media_and_publicity.md
- support.md
- sponsorship_and_donations.md
- Related Projects: related_projects.md
- Media and Publicity: media_and_publicity.md
- Support: support.md
- Sponsorship and Donations: sponsorship_and_donations.md
+27 -1
View File
@@ -1,5 +1,7 @@
/*
Copyright (c) 2016, Antonio SJ Musumeci <trapexit@spawn.link>
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
@@ -28,3 +30,27 @@ namespace assert
struct StaticAssert<true>
{};
}
#ifdef NDEBUG
#define ASSERT_NOT_NULL(ptr)
#else
#include <cassert>
#include <iostream>
#define ASSERT_NOT_NULL(ptr) \
do { \
auto *_tmp = (ptr); \
if((_tmp) == nullptr) { \
std::cerr << "ASSERT_NOT_NULL failed\n" \
<< " expression: \"" #ptr "\"\n" \
<< " file: " << __FILE__ << "\n" \
<< " line: " << __LINE__ << "\n" \
<< " function: " << __func__ << "\n"; \
assert((_tmp) != nullptr); \
} \
} while (0)
#endif
-10
View File
@@ -1,10 +0,0 @@
// Copyright David Abrahams 2002.
// Distributed under the Boost Software License, Version 1.0. (See
// accompanying file LICENSE_1_0.txt or copy at
// http://www.boost.org/LICENSE_1_0.txt)
#ifndef BOOST_WORKAROUND_DWA2002126_HPP
#define BOOST_WORKAROUND_DWA2002126_HPP
#include <boost/config/workaround.hpp>
#endif // BOOST_WORKAROUND_DWA2002126_HPP
-181
View File
@@ -1,181 +0,0 @@
// (C) Copyright John Maddock 2000.
// Use, modification and distribution are subject to the
// Boost Software License, Version 1.0. (See accompanying file
// LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
// See http://www.boost.org/libs/static_assert for documentation.
/*
Revision history:
02 August 2000
Initial version.
*/
#ifndef BOOST_STATIC_ASSERT_HPP
#define BOOST_STATIC_ASSERT_HPP
#include <boost/config.hpp>
#include <boost/detail/workaround.hpp>
#include <cstddef> //for std::size_t
#if defined(__GNUC__) && !defined(__GXX_EXPERIMENTAL_CXX0X__)
//
// This is horrible, but it seems to be the only we can shut up the
// "anonymous variadic macros were introduced in C99 [-Wvariadic-macros]"
// warning that get spewed out otherwise in non-C++11 mode.
//
#pragma GCC system_header
#endif
#ifndef BOOST_NO_CXX11_STATIC_ASSERT
# ifndef BOOST_NO_CXX11_VARIADIC_MACROS
# define BOOST_STATIC_ASSERT_MSG( ... ) static_assert(__VA_ARGS__)
# else
# define BOOST_STATIC_ASSERT_MSG( B, Msg ) static_assert( B, Msg )
# endif
#else
# define BOOST_STATIC_ASSERT_MSG( B, Msg ) BOOST_STATIC_ASSERT( B )
#endif
#ifdef BOOST_BORLANDC
//
// workaround for buggy integral-constant expression support:
#define BOOST_BUGGY_INTEGRAL_CONSTANT_EXPRESSIONS
#endif
#if defined(__GNUC__) && (__GNUC__ == 3) && ((__GNUC_MINOR__ == 3) || (__GNUC_MINOR__ == 4))
// gcc 3.3 and 3.4 don't produce good error messages with the default version:
# define BOOST_SA_GCC_WORKAROUND
#endif
//
// If the compiler issues warnings about old C style casts,
// then enable this:
//
#if defined(__GNUC__) && ((__GNUC__ > 3) || ((__GNUC__ == 3) && (__GNUC_MINOR__ >= 4)))
# ifndef BOOST_NO_CXX11_VARIADIC_MACROS
# define BOOST_STATIC_ASSERT_BOOL_CAST( ... ) ((__VA_ARGS__) != 0)
# else
# define BOOST_STATIC_ASSERT_BOOL_CAST( x ) ((x) != 0)
# endif
#else
# ifndef BOOST_NO_CXX11_VARIADIC_MACROS
# define BOOST_STATIC_ASSERT_BOOL_CAST( ... ) (bool)(__VA_ARGS__)
# else
# define BOOST_STATIC_ASSERT_BOOL_CAST(x) (bool)(x)
# endif
#endif
#ifndef BOOST_NO_CXX11_STATIC_ASSERT
# ifndef BOOST_NO_CXX11_VARIADIC_MACROS
# define BOOST_STATIC_ASSERT( ... ) static_assert(__VA_ARGS__, #__VA_ARGS__)
# else
# define BOOST_STATIC_ASSERT( B ) static_assert(B, #B)
# endif
#else
namespace boost{
// HP aCC cannot deal with missing names for template value parameters
template <bool x> struct STATIC_ASSERTION_FAILURE;
template <> struct STATIC_ASSERTION_FAILURE<true> { enum { value = 1 }; };
// HP aCC cannot deal with missing names for template value parameters
template<std::size_t x> struct static_assert_test{};
}
//
// Implicit instantiation requires that all member declarations be
// instantiated, but that the definitions are *not* instantiated.
//
// It's not particularly clear how this applies to enum's or typedefs;
// both are described as declarations [7.1.3] and [7.2] in the standard,
// however some compilers use "delayed evaluation" of one or more of
// these when implicitly instantiating templates. We use typedef declarations
// by default, but try defining BOOST_USE_ENUM_STATIC_ASSERT if the enum
// version gets better results from your compiler...
//
// Implementation:
// Both of these versions rely on sizeof(incomplete_type) generating an error
// message containing the name of the incomplete type. We use
// "STATIC_ASSERTION_FAILURE" as the type name here to generate
// an eye catching error message. The result of the sizeof expression is either
// used as an enum initialiser, or as a template argument depending which version
// is in use...
// Note that the argument to the assert is explicitly cast to bool using old-
// style casts: too many compilers currently have problems with static_cast
// when used inside integral constant expressions.
//
#if !defined(BOOST_BUGGY_INTEGRAL_CONSTANT_EXPRESSIONS)
#if defined(BOOST_MSVC) && defined(BOOST_NO_CXX11_VARIADIC_MACROS)
#define BOOST_STATIC_ASSERT( B ) \
typedef ::boost::static_assert_test<\
sizeof(::boost::STATIC_ASSERTION_FAILURE< BOOST_STATIC_ASSERT_BOOL_CAST ( B ) >)>\
BOOST_JOIN(boost_static_assert_typedef_, __COUNTER__)
#elif defined(BOOST_MSVC)
#define BOOST_STATIC_ASSERT(...) \
typedef ::boost::static_assert_test<\
sizeof(::boost::STATIC_ASSERTION_FAILURE< BOOST_STATIC_ASSERT_BOOL_CAST (__VA_ARGS__) >)>\
BOOST_JOIN(boost_static_assert_typedef_, __COUNTER__)
#elif (defined(BOOST_INTEL_CXX_VERSION) || defined(BOOST_SA_GCC_WORKAROUND)) && defined(BOOST_NO_CXX11_VARIADIC_MACROS)
// agurt 15/sep/02: a special care is needed to force Intel C++ issue an error
// instead of warning in case of failure
# define BOOST_STATIC_ASSERT( B ) \
typedef char BOOST_JOIN(boost_static_assert_typedef_, __LINE__) \
[ ::boost::STATIC_ASSERTION_FAILURE< BOOST_STATIC_ASSERT_BOOL_CAST( B ) >::value ]
#elif (defined(BOOST_INTEL_CXX_VERSION) || defined(BOOST_SA_GCC_WORKAROUND)) && !defined(BOOST_NO_CXX11_VARIADIC_MACROS)
// agurt 15/sep/02: a special care is needed to force Intel C++ issue an error
// instead of warning in case of failure
# define BOOST_STATIC_ASSERT(...) \
typedef char BOOST_JOIN(boost_static_assert_typedef_, __LINE__) \
[ ::boost::STATIC_ASSERTION_FAILURE< BOOST_STATIC_ASSERT_BOOL_CAST( __VA_ARGS__ ) >::value ]
#elif defined(__sgi)
// special version for SGI MIPSpro compiler
#define BOOST_STATIC_ASSERT( B ) \
BOOST_STATIC_CONSTANT(bool, \
BOOST_JOIN(boost_static_assert_test_, __LINE__) = ( B )); \
typedef ::boost::static_assert_test<\
sizeof(::boost::STATIC_ASSERTION_FAILURE< \
BOOST_JOIN(boost_static_assert_test_, __LINE__) >)>\
BOOST_JOIN(boost_static_assert_typedef_, __LINE__)
#elif BOOST_WORKAROUND(__MWERKS__, <= 0x3003)
// special version for CodeWarrior <= 8.x
#define BOOST_STATIC_ASSERT( B ) \
BOOST_STATIC_CONSTANT(int, \
BOOST_JOIN(boost_static_assert_test_, __LINE__) = \
sizeof(::boost::STATIC_ASSERTION_FAILURE< BOOST_STATIC_ASSERT_BOOL_CAST( B ) >) )
#else
// generic version
# ifndef BOOST_NO_CXX11_VARIADIC_MACROS
# define BOOST_STATIC_ASSERT( ... ) \
typedef ::boost::static_assert_test<\
sizeof(::boost::STATIC_ASSERTION_FAILURE< BOOST_STATIC_ASSERT_BOOL_CAST( __VA_ARGS__ ) >)>\
BOOST_JOIN(boost_static_assert_typedef_, __LINE__) BOOST_ATTRIBUTE_UNUSED
# else
# define BOOST_STATIC_ASSERT( B ) \
typedef ::boost::static_assert_test<\
sizeof(::boost::STATIC_ASSERTION_FAILURE< BOOST_STATIC_ASSERT_BOOL_CAST( B ) >)>\
BOOST_JOIN(boost_static_assert_typedef_, __LINE__) BOOST_ATTRIBUTE_UNUSED
# endif
#endif
#else
// alternative enum based implementation:
# ifndef BOOST_NO_CXX11_VARIADIC_MACROS
# define BOOST_STATIC_ASSERT( ... ) \
enum { BOOST_JOIN(boost_static_assert_enum_, __LINE__) \
= sizeof(::boost::STATIC_ASSERTION_FAILURE< (bool)( __VA_ARGS__ ) >) }
# else
# define BOOST_STATIC_ASSERT(B) \
enum { BOOST_JOIN(boost_static_assert_enum_, __LINE__) \
= sizeof(::boost::STATIC_ASSERTION_FAILURE< (bool)( B ) >) }
# endif
#endif
#endif // defined(BOOST_NO_CXX11_STATIC_ASSERT)
#endif // BOOST_STATIC_ASSERT_HPP
-70
View File
@@ -1,70 +0,0 @@
/* Hash function characterization.
*
* Copyright 2022-2024 Joaquin M Lopez Munoz.
* Distributed under the Boost Software License, Version 1.0.
* (See accompanying file LICENSE_1_0.txt or copy at
* http://www.boost.org/LICENSE_1_0.txt)
*
* See https://www.boost.org/libs/unordered for library home page.
*/
#ifndef BOOST_UNORDERED_HASH_TRAITS_HPP
#define BOOST_UNORDERED_HASH_TRAITS_HPP
#include <boost/unordered/detail/type_traits.hpp>
namespace boost{
namespace unordered{
namespace detail{
template<typename Hash,typename=void>
struct hash_is_avalanching_impl:std::false_type{};
template<typename IsAvalanching>
struct avalanching_value
{
static constexpr bool value=IsAvalanching::value;
};
/* may be explicitly marked as BOOST_DEPRECATED in the future */
template<> struct avalanching_value<void>
{
static constexpr bool value=true;
};
template<typename Hash>
struct hash_is_avalanching_impl<
Hash,
boost::unordered::detail::void_t<typename Hash::is_avalanching>
>:std::integral_constant<
bool,
avalanching_value<typename Hash::is_avalanching>::value
>{};
template<typename Hash>
struct hash_is_avalanching_impl<
Hash,
typename std::enable_if<((void)Hash::is_avalanching,true)>::type
>{}; /* Hash::is_avalanching is not a type: compile error downstream */
} /* namespace detail */
/* Each trait can be partially specialized by users for concrete hash functions
* when actual characterization differs from default.
*/
/* hash_is_avalanching<Hash>::value is:
* - false if Hash::is_avalanching is not present.
* - Hash::is_avalanching::value if this is present and constexpr-convertible
* to a bool.
* - true if Hash::is_avalanching is void (deprecated).
* - ill-formed otherwise.
*/
template<typename Hash>
struct hash_is_avalanching: detail::hash_is_avalanching_impl<Hash>::type{};
} /* namespace unordered */
} /* namespace boost */
#endif
+2 -10
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2018, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
@@ -17,11 +17,10 @@
*/
#include "branch.hpp"
#include "ef.hpp"
#include "errno.hpp"
#include "num.hpp"
Branch::Branch()
: mode(Branch::Mode::RW)
{
}
@@ -37,12 +36,6 @@ Branch::Branch(const u64 &default_minfreespace_)
{
}
int
Branch::from_string(const std::string_view str_)
{
return -EINVAL;
}
std::string
Branch::to_string(void) const
{
@@ -64,7 +57,6 @@ Branch::to_string(void) const
break;
}
if(std::holds_alternative<u64>(_minfreespace))
{
rv += ',';
+3 -6
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2018, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
@@ -20,7 +20,6 @@
#include "base_types.h"
#include "strvec.hpp"
#include "tofrom_string.hpp"
#include "fs_path.hpp"
#include <cstdint>
@@ -31,12 +30,11 @@
#include <variant>
class Branch final : public ToFromString
class Branch final
{
public:
enum class Mode
{
INVALID,
RO,
RW,
NC
@@ -61,8 +59,7 @@ public:
bool ro_or_nc(void) const;
public:
int from_string(const std::string_view str) final;
std::string to_string(void) const final;
std::string to_string(void) const;
public:
u64 minfreespace() const;
+114 -39
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2021, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
@@ -28,13 +28,15 @@
#include "str.hpp"
#include "syslog.hpp"
#include <string>
#include <mutex>
#include <optional>
#include <shared_mutex>
#include <string>
#include <fnmatch.h>
Branches::Impl::Impl(const u64 &default_minfreespace_)
Branches::Impl::Impl(const u64 *default_minfreespace_)
: _default_minfreespace(default_minfreespace_)
{
}
@@ -48,6 +50,16 @@ Branches::Impl::operator=(Branches::Impl &rval_)
*this_base = *rval_base;
// Branches using the default minfreespace hold a pointer to their
// owning Impl's _default_minfreespace. After copy/move those
// pointers still reference the source Impl, so re-link them to this
// Impl's _default_minfreespace.
for(auto &branch : *this)
{
if(std::holds_alternative<const u64*>(branch._minfreespace))
branch._minfreespace = _default_minfreespace;
}
return *this;
}
@@ -60,6 +72,16 @@ Branches::Impl::operator=(Branches::Impl &&rval_)
*this_base = std::move(*rval_base);
// Branches using the default minfreespace hold a pointer to their
// owning Impl's _default_minfreespace. After copy/move those
// pointers still reference the source Impl, so re-link them to this
// Impl's _default_minfreespace.
for(auto &branch : *this)
{
if(std::holds_alternative<const u64*>(branch._minfreespace))
branch._minfreespace = _default_minfreespace;
}
return *this;
}
@@ -67,7 +89,7 @@ const
u64&
Branches::Impl::minfreespace(void) const
{
return _default_minfreespace;
return *_default_minfreespace;
}
namespace l
@@ -82,7 +104,10 @@ namespace l
offset = s_.find_first_not_of("+<>-=");
if(offset == std::string::npos)
return;
{
*instr_ = s_.substr(0, std::min(s_.size(), (size_t)2));
return;
}
if(offset > 1)
offset = 2;
@@ -135,7 +160,7 @@ namespace l
std::string options;
std::vector<std::string> v;
str::rsplit1(str_,'=',&v);
v = str::rsplit1(str_,'=');
switch(v.size())
{
case 1:
@@ -145,8 +170,7 @@ namespace l
case 2:
*glob_ = v[0];
options = v[1];
v.clear();
str::split(options,',',&v);
v = str::split(options,',');
switch(v.size())
{
case 2:
@@ -227,9 +251,9 @@ namespace l
{
int rv;
StrVec paths;
Branches::Impl tmp_branches(branches_->minfreespace());
Branches::Impl tmp_branches(&branches_->minfreespace());
str::split(str_,':',&paths);
paths = str::split(str_,':');
for(auto &path : paths)
{
rv = l::parse(path,&tmp_branches);
@@ -249,9 +273,9 @@ namespace l
{
int rv;
std::vector<std::string> paths;
Branches::Impl tmp_branches(branches_->minfreespace());
Branches::Impl tmp_branches(&branches_->minfreespace());
str::split(str_,':',&paths);
paths = str::split(str_,':');
for(auto &path : paths)
{
rv = l::parse(path,&tmp_branches);
@@ -273,9 +297,9 @@ namespace l
{
int rv;
StrVec paths;
Branches::Impl tmp_branches(branches_->minfreespace());
Branches::Impl tmp_branches(&branches_->minfreespace());
str::split(str_,':',&paths);
paths = str::split(str_,':');
for(auto &path : paths)
{
rv = l::parse(path,&tmp_branches);
@@ -294,6 +318,9 @@ namespace l
int
erase_begin(Branches::Impl *branches_)
{
if(branches_->empty())
return -ENOENT;
branches_->erase(branches_->begin());
return 0;
@@ -303,6 +330,9 @@ namespace l
int
erase_end(Branches::Impl *branches_)
{
if(branches_->empty())
return -ENOENT;
branches_->pop_back();
return 0;
@@ -315,7 +345,7 @@ namespace l
{
StrVec patterns;
str::split(str_,':',&patterns);
patterns = str::split(str_,':');
for(auto i = branches_->begin(); i != branches_->end();)
{
int match = FNM_NOMATCH;
@@ -409,49 +439,94 @@ Branches::from_string(const std::string_view str_)
Branches::Ptr impl;
Branches::Ptr new_impl;
{
std::lock_guard<std::mutex> lock_guard(_mutex);
impl = _impl;
}
new_impl = std::make_shared<Branches::Impl>(&minfreespace);
new_impl = std::make_shared<Branches::Impl>(impl->minfreespace());
*new_impl = *impl;
for(;;)
{
{
std::shared_lock<std::shared_mutex> lk(_mutex);
impl = _impl;
}
rv = new_impl->from_string(str_);
if(rv < 0)
return rv;
*new_impl = *impl;
{
std::lock_guard<std::mutex> lock_guard(_mutex);
_impl = new_impl;
}
rv = new_impl->from_string(str_);
if(rv < 0)
return rv;
return 0;
{
std::unique_lock<std::shared_mutex> lk(_mutex);
if(impl != _impl)
continue;
_impl = std::move(new_impl);
}
return 0;
}
}
std::string
Branches::to_string(void) const
{
std::lock_guard<std::mutex> lock_guard(_mutex);
Branches::Ptr impl;
return _impl->to_string();
{
std::shared_lock<std::shared_mutex> lk(_mutex);
impl = _impl;
}
return impl->to_string();
}
void
Branches::find_and_set_mode_ro()
{
for(auto &branch : *_impl)
Branches::Ptr impl;
Branches::Ptr new_impl;
std::vector<fs::path> ro_paths;
new_impl = std::make_shared<Branches::Impl>(&minfreespace);
for(;;)
{
if(branch.mode != Branch::Mode::RW)
continue;
ro_paths.clear();
if(!fs::is_rofs_but_not_mounted_ro(branch.path))
continue;
{
std::shared_lock<std::shared_mutex> lk(_mutex);
impl = _impl;
}
SysLog::warning("branch `{}` found to be readonly - setting its mode=RO",
branch.path.string());
*new_impl = *impl;
branch.mode = Branch::Mode::RO;
for(auto &branch : *new_impl)
{
if(branch.mode != Branch::Mode::RW)
continue;
if(!fs::is_rofs_but_not_mounted_ro(branch.path))
continue;
branch.mode = Branch::Mode::RO;
ro_paths.emplace_back(branch.path);
}
if(ro_paths.empty())
return;
{
std::unique_lock<std::shared_mutex> lk(_mutex);
if(impl != _impl)
continue;
_impl = std::move(new_impl);
}
for(const auto &path : ro_paths)
SysLog::warning("branch `{}` found to be readonly - setting its mode=RO",
path.string());
return;
}
}
+17 -16
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2021, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
@@ -25,10 +25,9 @@
#include <cstdint>
#include <memory>
#include <mutex>
#include <shared_mutex>
#include <string>
#include <vector>
#include <memory>
#define MINFREESPACE_DEFAULT (4294967295ULL)
@@ -40,13 +39,13 @@ public:
public std::vector<Branch>
{
private:
const u64 &_default_minfreespace;
const u64 *_default_minfreespace;
public:
using Ptr = std::shared_ptr<Impl>;
public:
Impl(const u64 &default_minfreespace);
Impl(const u64 *default_minfreespace);
public:
int from_string(const std::string_view str) final;
@@ -70,12 +69,12 @@ public:
u64 minfreespace = MINFREESPACE_DEFAULT;
private:
mutable std::mutex _mutex;
Ptr _impl;
Branches::Ptr _impl;
mutable std::shared_mutex _mutex;
public:
Branches()
: _impl(std::make_shared<Impl>(minfreespace))
: _impl(std::make_shared<Impl>(&minfreespace))
{
}
@@ -84,15 +83,17 @@ public:
std::string to_string(void) const final;
public:
operator Ptr() const { std::lock_guard<std::mutex> lg(_mutex); return _impl; }
Ptr operator->() const { std::lock_guard<std::mutex> lg(_mutex); return _impl; }
public:
Impl::iterator begin() { return _impl->begin(); }
Impl::iterator end() { return _impl->end(); }
Impl::const_iterator begin() const { return _impl->begin(); }
Impl::const_iterator end() const { return _impl->end(); }
operator Ptr() const
{
std::shared_lock<std::shared_mutex> lk(_mutex);
return _impl;
}
Ptr operator->() const
{
std::shared_lock<std::shared_mutex> lk(_mutex);
return _impl;
}
public:
void find_and_set_mode_ro();
+3 -1
View File
@@ -1,5 +1,7 @@
/*
Copyright (c) 2016, Antonio SJ Musumeci <trapexit@spawn.link>
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+3 -1
View File
@@ -1,5 +1,7 @@
/*
Copyright (c) 2016, Antonio SJ Musumeci <trapexit@spawn.link>
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+18
View File
@@ -1,3 +1,21 @@
/*
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include "caps.hpp"
#if defined __linux__
+18
View File
@@ -1,3 +1,21 @@
/*
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#pragma once
namespace caps
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2020, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+3 -1
View File
@@ -1,5 +1,7 @@
/*
Copyright (c) 2020, Antonio SJ Musumeci <trapexit@spawn.link>
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+42 -83
View File
@@ -1,5 +1,7 @@
/*
Copyright (c) 2016, Antonio SJ Musumeci <trapexit@spawn.link>
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
@@ -15,35 +17,27 @@
*/
#include "config.hpp"
#include "ef.hpp"
#include "errno.hpp"
#include "from_string.hpp"
#include "fmt/core.h"
#include "fs_path.hpp"
#include "nonstd/string.hpp"
#include "num.hpp"
#include "rwlock.hpp"
#include "str.hpp"
#include "to_string.hpp"
#include "version.hpp"
#include <algorithm>
#include <cstdint>
#include <fstream>
#include <iostream>
#include <string>
#include <pthread.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
constexpr static const char CACHE_FILES_PROCESS_NAMES_DEFAULT[] =
constexpr const char CACHE_FILES_PROCESS_NAMES_DEFAULT[] =
"rtorrent|"
"qbittorrent-nox";
Config cfg;
thread_local int Config::CfgConfigFile::_depth = 0;
Config::CfgConfigFile::CfgConfigFile()
{
}
@@ -57,6 +51,7 @@ Config::CfgConfigFile::from_string(const std::string_view s_)
if(_depth > 5)
return -ELOOP;
_depth++;
struct DepthGuard { int &d; ~DepthGuard() { --d; } } guard{_depth};
cfg_file = (s_.empty() ? _cfg_file : s_);
@@ -64,8 +59,6 @@ Config::CfgConfigFile::from_string(const std::string_view s_)
if(rv == 0)
_cfg_file = cfg_file;
_depth--;
return rv;
}
@@ -110,7 +103,7 @@ Config::Config()
lazy_umount_mountpoint(false),
link_cow(false),
link_exdev(LinkEXDEV::ENUM::PASSTHROUGH),
log_metrics(false),
log_file({}),
minfreespace(branches.minfreespace),
mountpoint(),
moveonenospc(true),
@@ -137,15 +130,11 @@ Config::Config()
xattr(XAttr::ENUM::PASSTHROUGH),
_congestion_threshold(fuse_cfg.congestion_threshold),
_debug(fuse_cfg.debug),
_debug(false),
_gid(fuse_cfg.gid),
_max_background(fuse_cfg.max_background),
_mount(mountpoint),
_mountpoint(mountpoint),
_never_forget_nodes(),
_noforget(),
_remember(fuse_cfg.remember_nodes),
_remember_nodes(fuse_cfg.remember_nodes),
_srcmounts(branches),
_threads(fuse_cfg.read_thread_count),
_uid(fuse_cfg.uid),
@@ -154,13 +143,22 @@ Config::Config()
_initialized(false)
{
allow_idmap.ro =
_congestion_threshold.ro =
_max_background.ro =
_mount.ro =
_mountpoint.ro =
_never_forget_nodes.ro =
_noforget.ro =
_remember.ro =
_remember_nodes.ro =
_srcmounts.ro =
_version.ro =
allow_idmap.ro =
async_read.ro =
branches_mount_timeout.ro =
branches_mount_timeout_fail.ro =
cache_symlinks.ro =
cache_writeback.ro =
_congestion_threshold.ro =
direct_io_allow_mmap.ro =
export_support.ro =
fsname.ro =
@@ -168,9 +166,6 @@ Config::Config()
handle_killpriv.ro =
handle_killpriv_v2.ro =
kernel_permissions_check.ro =
_max_background.ro =
_mount.ro =
_mountpoint.ro =
nullrw.ro =
pid.ro =
pin_threads.ro =
@@ -179,8 +174,6 @@ Config::Config()
process_thread_queue_depth.ro =
read_thread_count.ro =
scheduling_priority.ro =
_srcmounts.ro =
_version.ro =
true;
_congestion_threshold.display =
_gid.display =
@@ -188,6 +181,7 @@ Config::Config()
_mount.display =
_noforget.display =
_remember.display =
_remember_nodes.display =
_srcmounts.display =
_threads.display =
_uid.display =
@@ -260,7 +254,7 @@ Config::Config()
_map["lazy-umount-mountpoint"] = &lazy_umount_mountpoint;
_map["link-cow"] = &link_cow;
_map["link-exdev"] = &link_exdev;
_map["log.metrics"] = &log_metrics;
_map["log.file"] = &log_file;
_map["minfreespace"] = &minfreespace;
_map["mount"] = &_mount;
_map["mountpoint"] = &_mountpoint;
@@ -312,36 +306,6 @@ Config::has_key(const std::string &key_) const
return _map.count(key_);
}
void
Config::keys(std::string &s_) const
{
s_.reserve(512);
for(const auto &[key,val] : _map)
{
s_ += key;
s_ += '\0';
}
s_.resize(s_.size() - 1);
}
void
Config::keys_xattr(std::string &s_) const
{
s_.reserve(1024);
for(const auto &[key,val] : _map)
{
if(val->display == false)
continue;
s_ += "user.mergerfs.";
s_ += key;
s_ += '\0';
}
}
ssize_t
Config::keys_listxattr_size() const
{
@@ -375,13 +339,16 @@ Config::keys_listxattr(char *list_,
if(val->display == false)
continue;
auto rv = fmt::format_to_n(list,size,
"user.mergerfs.{}\0",
key);
if(rv.out > (list + size))
static constexpr std::string_view prefix = "user.mergerfs.";
ssize_t entry_size = (ssize_t)(prefix.size() + key.size() + 1);
if(entry_size > size)
return -ERANGE;
list += rv.size;
size -= rv.size;
memcpy(list, prefix.data(), prefix.size());
list += prefix.size();
memcpy(list, key.data(), key.size());
list += key.size();
*list++ = '\0';
size -= entry_size;
}
return (list - list_);
@@ -427,10 +394,7 @@ Config::set(const std::string &key_,
int
Config::set(const std::string &kv_)
{
std::string key;
std::string val;
str::splitkv(kv_,'=',&key,&val);
auto [key, val] = str::splitkv(kv_,'=');
key = str::trim(key);
val = str::trim(val);
@@ -440,7 +404,6 @@ Config::set(const std::string &kv_)
int
Config::from_stream(std::istream &istrm_)
{
int rv;
std::string line;
Config::ErrVec new_errs;
@@ -450,37 +413,33 @@ Config::from_stream(std::istream &istrm_)
if(line.empty() || (line[0] == '#'))
continue;
rv = set(line);
int rv = set(line);
if(rv < 0)
new_errs.push_back({-rv,line});
}
rv = (new_errs.empty() ? 0 : -EINVAL);
errs.insert(errs.end(),
new_errs.begin(),
new_errs.end());
return rv;
return (new_errs.empty() ? 0 : -EINVAL);
}
int
Config::from_file(const std::string &filepath_)
{
int rv;
std::ifstream ifstrm;
errno = 0;
ifstrm.open(filepath_);
if(!ifstrm.good())
{
errno = errno ? errno : EIO;
errs.push_back({errno,filepath_});
return -errno;
}
rv = from_stream(ifstrm);
ifstrm.close();
return rv;
return from_stream(ifstrm);
}
void
@@ -510,13 +469,13 @@ Config::is_mergerfs_xattr(const char *attrname_)
bool
Config::is_cmd_xattr(const std::string_view &attrname_)
{
return nonstd::string::starts_with(attrname_,"user.mergerfs.cmd.");
return nonstd::starts_with(attrname_,"user.mergerfs.cmd.");
}
std::string
Config::prune_ctrl_xattr(const std::string &s_)
std::string_view
Config::prune_ctrl_xattr(const std::string_view s_)
{
const size_t offset = (sizeof("user.mergerfs.") - 1);
constexpr size_t offset = (sizeof("user.mergerfs.") - 1);
if(offset < s_.size())
return s_.substr(offset);
+22 -26
View File
@@ -1,5 +1,7 @@
/*
Copyright (c) 2016, Antonio SJ Musumeci <trapexit@spawn.link>
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
@@ -19,12 +21,13 @@
#include "branches.hpp"
#include "category.hpp"
#include "config_cachefiles.hpp"
#include "config_debug.hpp"
#include "config_dummy.hpp"
#include "config_flushonclose.hpp"
#include "config_follow_symlinks.hpp"
#include "config_inodecalc.hpp"
#include "config_link_exdev.hpp"
#include "config_log_metrics.hpp"
#include "config_log_file.hpp"
#include "config_moveonenospc.hpp"
#include "config_nfsopenhack.hpp"
#include "config_noforget.hpp"
@@ -44,14 +47,14 @@
#include "fuse_cfg.hpp"
#include "fuse_readdir.hpp"
#include "policy.hpp"
#include "rwlock.hpp"
#include "tofrom_ref.hpp"
#include "tofrom_wrapper.hpp"
#include "syslog.hpp"
#include "fuse.h"
#include <cstdint>
#include "base_types.h"
#include <map>
#include <memory>
#include <string>
@@ -60,15 +63,14 @@
#include <sys/stat.h>
typedef ToFromWrapper<bool> ConfigBOOL;
typedef ToFromWrapper<uint64_t> ConfigUINT64;
typedef ToFromWrapper<int64_t> ConfigS64;
typedef ToFromWrapper<u64> ConfigU64;
typedef ToFromWrapper<s64> ConfigS64;
typedef ToFromWrapper<int> ConfigINT;
typedef ToFromWrapper<std::string> ConfigSTR;
typedef ToFromWrapper<fs::path> ConfigPath;
typedef std::map<std::string,ToFromString*> Str2TFStrMap;
typedef ROToFromWrapper<std::string> ConfigROSTR;
extern const std::string CONTROLFILE;
class Config
{
@@ -91,7 +93,7 @@ public:
{
private:
fs::path _cfg_file;
int _depth = 0;
static thread_local int _depth;
public:
CfgConfigFile();
@@ -104,22 +106,19 @@ public:
public:
Config();
public:
Config& operator=(const Config&);
public:
ConfigBOOL allow_idmap;
ConfigBOOL async_read;
Branches branches;
ConfigUINT64 branches_mount_timeout;
ConfigU64 branches_mount_timeout;
ConfigBOOL branches_mount_timeout_fail;
ConfigUINT64 cache_attr;
ConfigUINT64 cache_entry;
ConfigU64 cache_attr;
ConfigU64 cache_entry;
CacheFiles cache_files;
ConfigSet cache_files_process_names;
ConfigUINT64 cache_negative_entry;
ConfigU64 cache_negative_entry;
ConfigBOOL cache_readdir;
ConfigUINT64 cache_statfs;
ConfigU64 cache_statfs;
ConfigBOOL cache_symlinks;
ConfigBOOL cache_writeback;
Categories category;
@@ -140,7 +139,7 @@ public:
ConfigBOOL lazy_umount_mountpoint;
ConfigBOOL link_cow;
LinkEXDEV link_exdev;
LogMetrics log_metrics;
LogFile log_file;
TFSRef<u64> minfreespace;
fs::path mountpoint;
MoveOnENOSPC moveonenospc;
@@ -156,7 +155,7 @@ public:
TFSRef<int> process_thread_queue_depth;
ProxyIOPrio proxy_ioprio;
TFSRef<int> read_thread_count;
ConfigUINT64 readahead;
ConfigU64 readahead;
FUSE::ReadDir readdir;
RenameEXDEV rename_exdev;
ConfigINT scheduling_priority;
@@ -169,7 +168,7 @@ public:
private:
TFSRef<int> _congestion_threshold;
TFSRef<bool> _debug;
Debug _debug;
CfgDummy _dummy;
TFSRef<s64> _gid;
TFSRef<int> _max_background;
@@ -177,8 +176,8 @@ private:
TFSRef<fs::path> _mountpoint;
CfgNoforget _never_forget_nodes;
CfgNoforget _noforget;
TFSRef<s64> _remember;
TFSRef<s64> _remember_nodes;
CfgNoforget _remember;
CfgNoforget _remember_nodes;
SrcMounts _srcmounts;
TFSRef<int> _threads;
TFSRef<s64> _uid;
@@ -197,10 +196,9 @@ public:
public:
bool has_key(const std::string &key) const;
void keys(std::string &s) const;
void keys_xattr(std::string &s) const;
ssize_t keys_listxattr(char *list, size_t size) const;
ssize_t keys_listxattr_size() const;
const Str2TFStrMap& get_map() const { return _map; }
public:
int get(const std::string &key, std::string *val) const;
@@ -216,13 +214,11 @@ public:
static bool is_ctrl_file(const fs::path &fusepath);
static bool is_mergerfs_xattr(const char *attrname);
static bool is_cmd_xattr(const std::string_view &attrname);
static std::string prune_ctrl_xattr(const std::string &s);
static std::string_view prune_ctrl_xattr(const std::string_view s);
static std::string_view prune_cmd_xattr(const std::string_view &s);
private:
Str2TFStrMap _map;
};
std::ostream& operator<<(std::ostream &s,const Config::ErrVec &ev);
extern Config cfg;
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2020, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2020, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+63
View File
@@ -0,0 +1,63 @@
/*
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include "config_debug.hpp"
#include "debug.hpp"
#include "from_string.hpp"
#include "fuse_cfg.hpp"
#include "to_string.hpp"
#include <memory>
#include <string>
Debug::Debug(const bool v_)
{
fuse_cfg.debug = v_;
}
std::string
Debug::to_string(void) const
{
return str::to(fuse_cfg.debug);
}
int
Debug::from_string(const std::string_view s_)
{
int rv;
bool val;
rv = str::from(s_,&val);
if(rv < 0)
return rv;
fuse_cfg.debug = val;
// If debug is being enabled, ensure output file is set
if(val)
{
auto log_file_path = fuse_cfg.log_filepath();
if(log_file_path && !log_file_path->empty())
fuse_debug_set_output(*log_file_path);
else
fuse_debug_set_output({}); // Use stderr
}
return 0;
}
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2021, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
@@ -20,10 +20,10 @@
#include "tofrom_string.hpp"
class LogMetrics : public ToFromString
class Debug : public ToFromString
{
public:
LogMetrics(const bool);
Debug(const bool);
public:
std::string to_string(void) const final;
+18
View File
@@ -1,3 +1,21 @@
/*
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#pragma once
#include "tofrom_string.hpp"
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2023, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2023, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2020, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2020, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2020, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2020, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2021, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
+1 -1
View File
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2021, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
@@ -1,7 +1,7 @@
/*
ISC License
Copyright (c) 2021, Antonio SJ Musumeci <trapexit@spawn.link>
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
@@ -16,38 +16,32 @@
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include "config_log_metrics.hpp"
#include "from_string.hpp"
#include "to_string.hpp"
#include "config_log_file.hpp"
#include "fuse.h"
#include "debug.hpp"
#include "fuse_cfg.hpp"
LogMetrics::LogMetrics(const bool val_)
#include <memory>
#include <string>
LogFile::LogFile(const std::string &s_)
{
fuse_log_metrics_set(val_);
if(!s_.empty())
fuse_debug_set_output(s_);
}
std::string
LogMetrics::to_string(void) const
LogFile::to_string(void) const
{
bool val;
auto filepath = fuse_cfg.log_filepath();
val = fuse_log_metrics_get();
return str::to(val);
return (filepath ? *filepath : "");
}
int
LogMetrics::from_string(const std::string_view s_)
LogFile::from_string(const std::string_view s_)
{
int rv;
bool val;
const std::string s{s_};
rv = str::from(s_,&val);
if(rv < 0)
return rv;
fuse_log_metrics_set(val);
return 0;
return fuse_debug_set_output(s);
}

Some files were not shown because too many files have changed in this diff Show More