Compare commits

...
29 Commits
Author SHA1 Message Date
natalefalco 8ea47df13e Include Docker Compose setup for mergerfs
Added Docker Compose configuration for running mergerfs.
2026-07-31 08:13:59 -05:00
shyney7 42ab831d97 docs/setup: link install :z example to SELinux FAQ 2026-05-17 10:14:21 -05:00
shyney7 89b4046d11 docs/faq: add SELinux relabeling entry 2026-05-17 10:14:21 -05:00
Antonio SJ Musumeci 7ec6f8e2f2 Manage cloned /dev/fuse fds in sessions
- Store cloned read-worker fds on fuse_session and close them there.
- Set up clone fallback through session helpers used by the loop.
- Carry per-request fds through receive, process, and reply paths.

Tests show significant performance increases in highly concurrent
workloads on a high core count system. Upto 15% better on reads and
27% on writes.
2026-05-14 20:13:07 -05:00
Antonio SJ Musumeci 5f7fa17ad8 Inline ioprio disabled fast path
- Expose the ioprio enabled flag as an atomic for relaxed inline checks
- Inline SetFrom construction so disabled read/write paths avoid a function call
- Move priority application into a slow helper used only when ioprio is enabled
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 30d59d4458 Use shared_mutex for fuse log config
- Guard fuse_cfg_t log file and filepath shared pointers with a shared mutex
- Replace atomic shared pointer loads and stores with shared and unique locks
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 02dc36080c Use shared_mutex for ReadDir state
- Replace atomic shared pointer loads/stores with shared/unique locking.
- Store the configured readdir mode directly as std::string.
- Copy the active ReadDir implementation under lock before invoking it.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 8734847968 Guard branch snapshots with shared_mutex
Important changes:
- Replace atomic shared_ptr operations with shared_mutex-protected snapshots.
- Retry copy-on-write branch updates when another writer publishes first.
- Keep long-running parsing and read-only filesystem checks outside branch locks.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 608cd00764 Use atomic open file reference counts
- Use relaxed increment under the map bucket lock when duplicating file handles.
- Use acq_rel decrement when releasing and checking for final references.
- Load moved ref_count with relaxed ordering during map insertion.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci b13727abde Document readdir merge mutex scope
- Explain that the mutex is per readdir call.
- Document that only dirent deduplication and list append are serialized.
- Clarify that per-branch opendir and getdents64 work runs outside the critical section.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 231aac15cd Use shared write locks until ENOSPC relocation
- Replace the per-file Mutex with std::shared_mutex for shared write access.
- Run direct and cached writes under a shared lock while the fd is stable.
- Retry failed writes under a unique lock before move-on-ENOSPC relocation.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 8b17d5f35e Synchronize node updates with path release
- Add helpers that release path locks and update node state under f.lock.
- Update getattr and setattr stat-cache invalidation while preserving ESTALE handling.
- Rework create, open, tmpfile, and set_path_info to mutate lookup/open state under the shared lock.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 8608b2be46 Clarify releasedir readdir synchronization barrier
- Document the lock/unlock pair as an acquire/release barrier.
- Explain that releasedir waits for in-flight readdir calls before destroy/free.
- Warn against removing the synchronization despite it not being a critical section.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 1e0bd54989 Move statvfs cache to use shared mutex
Its read heavy and micro benchmarks show no regression on the low end
and 30%+ throughput improvements on the highend.
2026-05-14 00:22:06 -04:00
Antonio SJ Musumeci 56285fc6e5 Pass annobin override to RPM toolset builds
Important changes:
- Add Makefile RPMBUILD and RPMBUILD_FLAGS overrides for the rpm target.
- Detect /opt/rh GCC toolsets before RPM packaging.
- Pass --undefine _annotated_build to avoid Rocky 8 annobin plugin failures.
2026-05-13 17:01:41 -04:00
Antonio SJ Musumeci dfe053a0ae Require C++20-capable build toolchains
Important changes:
- Switch mergerfs and vendored libfuse Makefiles to -std=c++20.
- Select the latest available GCC/G++ or toolset in container builds.
- Verify compiler C++20 support during package installation.
- Fix dynamic SysLog formatting for fmt C++20 checks.
2026-05-13 13:55:43 -05:00
Antonio SJ Musumeci 12ac368e8b Preserve CPPFLAGS in release builds
- Append -DNDEBUG to existing CPPFLAGS in the release target.
- Keep caller-provided build flags for Debian packaging.
2026-05-09 19:28:56 -05:00
Antonio SJ Musumeci c8f32f9f25 update build system and dependency configuration 2026-05-09 18:55:23 -05:00
Antonio SJ Musumeci ce3737ff33 Add new OS build files 2026-05-07 23:35:14 -04:00
Antonio SJ Musumeci 84e4cdeb7e make all integration tests self-contained with ephemeral mounts
- Add mergerfs_mount() context manager in posix_parity.py that creates
  temporary branch dirs in tests/.test_tmp/, mounts an ephemeral mergerfs
  instance, and cleans up after use.
- Add find_mergerfs(), find_fusermount(), mount_mergerfs(), unmount_mergerfs()
  helpers.
- Update run-tests to invoke each TEST_* script without a mountpoint arg.
- Update README.md to document the new `python3 tests/run-tests` invocation.
- Fix 44 tests to use mergerfs_mount() instead of expecting a pre-mounted path.
- Add tempfile import to 10 tests that were missing it.
- Fix TEST_mount_lifecycle to use mergerfs_mount() context manager.
- Fix TEST_posix_tmpfile missing join import.
- Fix TEST_cfg_xattr_modes error message framing (no "(known bug)").
- Fix TEST_posix_copy_file_range: loop-copy full size, compare st_size and
  SHA256 hash; add NativeTmp for space management; route native temp to
  tests/.test_tmp/ to avoid ENOSPC on /tmp.
- Fix TEST_posix_getattr_fgetattr: st_cmp skips st_size for dirs; add
  compare_ino param for safe inode comparison when inodecalc=passthrough.
- Fix TEST_unlink_rename, TEST_use_* tests: namespace cleanup, consistent
  variable naming, None check for cmpfunc.
- Fix TEST_readlink_semantics: remove type annotations to support older
  Python; wrap in mergerfs_mount() try/except.
- Add flush=True to run-tests print so test names show immediately.
2026-05-07 21:47:40 -04:00
Antonio SJ Musumeci 6d6d73c7b3 Use unmount() on FreeBSD for ENOTCONN umount 2026-05-07 17:49:24 -04:00
Antonio SJ Musumeci bea31543a0 update fuse_lib_tmpfile() node management to match fuse_lib_create() 2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci 99835a0178 Document advisory locking limitation and update deprecated options
- Add documentation explaining that advisory file locking is not supported

- Add remember-nodes to deprecated options mapping

- Add comment explaining stat_crc32b fingerprint invalidation behavior

- Fix heading level for Host vs Container identity section
2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci 342a1efcfb Add QEMU binfmt setup and fix static installer
- Fix typo in static.installer: /pkg/ -> /pkgs/

- Add missing mkdir -p /build before makeself

- Mount pkgs directory via podman build -v so static.installer can access

  previously built static tarballs without relying on bind mounts relative

  to the build context

- Register qemu-user-static binfmt handlers for cross-compilation

- Make --git-repo optional when using --setup
2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci c587a127fb add missing <new> include for std::nothrow in fuse_session.cpp 2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci f1034a3ad6 Remove timed node removal 2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci 8117998205 Remove unused lock operation plumbing 2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci cc780bc521 libfuse: remove per-node lock bookkeeping 2026-05-06 20:49:46 -04:00
Antonio SJ Musumeci 14b7c8e607 Use stat fingerprint as cache validity sentinel
- Reserve a zero stat CRC to represent an invalid cache fingerprint.

- Include device, size, inode, mtime, and ctime data in the fingerprint.

- Recompute the fingerprint during auto-cache open before keeping cache.
2026-05-06 20:49:46 -04:00
112 changed files with 3956 additions and 3810 deletions
+31 -22
View File
@@ -29,6 +29,8 @@ MKDIR ?= mkdir
MKTEMP ?= mktemp
MV ?= mv
RM ?= rm
RPMBUILD ?= rpmbuild
RPMBUILD_FLAGS ?=
SED ?= sed
STRIP ?= strip
TAR ?= tar
@@ -37,7 +39,7 @@ TOUCH ?= touch
BUILDDIR := build
GITHUB_REPO ?= "https://github.com/trapexit/mergerfs"
DEFAULT_RELEASE_SAMPLE := debian:13.amd64
DEFAULT_TARGET := debian:13.amd64
ifndef GIT_REPO
ifneq ($(shell $(GIT) --version 2> /dev/null),)
@@ -48,11 +50,7 @@ endif
endif
USE_XATTR ?= 1
UGID_USE_RWLOCK ?= 0
ifdef NDEBUG
OPT_FLAGS := -O2 -DNDEBUG
else
ifdef SANITIZE
ifeq ($(SANITIZE),1)
override SANITIZE := -fsanitize=address,undefined,leak
@@ -60,14 +58,22 @@ else ifeq ($(filter -fsanitize=%,$(SANITIZE)),)
override SANITIZE := -fsanitize=$(SANITIZE)
endif
endif
OPT_FLAGS := -O0 \
-g \
-fno-omit-frame-pointer \
$(SANITIZE) \
-fstack-protector-strong \
-Wextra \
-Wno-unused-parameter \
-DDEBUG
COMMON_OPT_FLAGS += \
-fstack-protector-strong \
-Wextra \
-Wno-unused-parameter
DEBUG_OPT_FLAGS += \
-O0 \
-g3 \
-fno-omit-frame-pointer \
$(SANITIZE)
ifdef RELEASE
OPT_FLAGS := -O3 $(COMMON_OPT_FLAGS)
else
OPT_FLAGS ?= $(DEBUG_OPT_FLAGS) $(COMMON_OPT_FLAGS)
endif
ifdef STATIC
@@ -117,7 +123,7 @@ CXXFLAGS ?= \
-Wno-unused-result \
-pipe
override CXXFLAGS += \
-std=c++17 \
-std=c++20 \
-MMD \
-MP
override INC_FLAGS := \
@@ -125,8 +131,7 @@ override INC_FLAGS := \
-Ivendored \
-Ivendored/libfuse/include
override MFS_FLAGS := \
-DUSE_XATTR=$(USE_XATTR) \
-DUGID_USE_RWLOCK=$(UGID_USE_RWLOCK)
-DUSE_XATTR=$(USE_XATTR)
override TESTS_FLAGS := \
-Isrc \
-Ivendored \
@@ -198,7 +203,7 @@ libfuse: $(LIBFUSE)
$(MAKE) -C vendored/libfuse
$(LIBFUSE):
$(MAKE) -C vendored/libfuse $(BUILDDIR)/libfuse.a
$(MAKE) -C vendored/libfuse $(BUILDDIR)/libfuse.a RELEASE=$(RELEASE)
tests: $(BUILDDIR)/tests
@@ -340,7 +345,7 @@ rpm: tarball
$(SED) 's/__VERSION__/$(VERSION)/g' mergerfs.spec > \
rpmbuild/SOURCES/mergerfs.spec
cp -ar mergerfs-$(VERSION).tar.gz rpmbuild/SOURCES
rpmbuild -ba rpmbuild/SOURCES/mergerfs.spec \
$(RPMBUILD) $(RPMBUILD_FLAGS) -ba rpmbuild/SOURCES/mergerfs.spec \
--define "_topdir $(CURDIR)/rpmbuild"
.PHONY: install-build-pkgs
@@ -362,14 +367,16 @@ define build_release
--branch=$(GITREF)
endef
.PHONY: release
.PHONY: release release-all
.PHONY: release-amd64 release-arm64 release-armhf release-riscv64
.PHONY: release-sample release-static release-tarball
.PHONY: release-sample release-static release-tarball release-installer
release:
$(MAKE) RELEASE=1 CPPFLAGS="$(CPPFLAGS) -DNDEBUG" all
release-all:
$(call build_release,"all")
release-sample:
$(if $(RELEASE_SAMPLE),,$(eval RELEASE_SAMPLE := $(shell ./buildtools/detect-local-target 2>/dev/null)))
$(call build_release,$(if $(RELEASE_SAMPLE),$(RELEASE_SAMPLE),$(DEFAULT_RELEASE_SAMPLE)))
$(if $(TARGET),,$(eval TARGET := $(shell ./buildtools/detect-local-target 2>/dev/null)))
$(call build_release,$(if $(TARGET),$(TARGET),$(DEFAULT_TARGET)))
release-amd64:
$(call build_release,"amd64")
release-arm64:
@@ -382,6 +389,8 @@ release-static:
$(call build_release,"static")
release-tarball:
$(call build_release,"tarball")
release-installer:
$(call build_release,"installer")
container:
$(eval GITREF ?= $(shell git describe --exact-match --tags HEAD 2>/dev/null || git symbolic-ref --short HEAD 2>/dev/null || git rev-parse --short HEAD))
+15 -3
View File
@@ -1,6 +1,19 @@
#!/bin/sh
set -e
if [ -e /tmp/build-env ]; then
. /tmp/build-env
fi
rpmbuild_flags()
{
case "$(command -v "${CXX:-g++}")" in
/opt/rh/*)
printf '%s' '--undefine _annotated_build'
;;
esac
}
if [ $# -ne 2 ]; then
echo "usage: $0 <branch> <git_repo>"
echo "argv: $@"
@@ -28,10 +41,9 @@ mkdir "/build"
if [ -e /usr/bin/apt-get ]; then
make deb
elif [ -e /usr/bin/dnf ]; then
make rpm
make RPMBUILD_FLAGS="$(rpmbuild_flags)" rpm
elif [ -e /usr/bin/yum ]; then
. /opt/rh/devtoolset-9/enable
make rpm
make RPMBUILD_FLAGS="$(rpmbuild_flags)" rpm
elif [ -e /sbin/apk ]; then
echo "NOT YET SUPPORTED"
exit 1
+32 -4
View File
@@ -18,10 +18,12 @@ def build(git_repo,
'--force-rm']
if os.path.exists(git_repo):
git_repo = os.path.realpath(git_repo)
args += ['-v',f'{git_repo}:/tmp/mergerfs.git:ro']
args += ['-v',f'{git_repo}:/tmp/mergerfs.git:ro:z']
args += ['--build-arg=GIT_REPO=file:///tmp/mergerfs.git']
else:
args += [f'--build-arg=GIT_REPO={git_repo}']
os.makedirs(pkgdirpath,exist_ok=True)
args += ['-v',f'{os.path.realpath(pkgdirpath)}:/pkgs:ro']
args += ['-o',pkgdirpath,
'-f',containerfile,
f'--build-arg=BRANCH={branch}',
@@ -30,8 +32,6 @@ def build(git_repo,
# TODO: Capture output and write to log
print(args)
rv = subprocess.run(args)
os.makedirs(pkgdirpath,exist_ok=True)
report_filepath = os.path.join(pkgdirpath,"build-report.txt")
with open(report_filepath,"a+") as f:
build = os.path.basename(containerfile)
@@ -66,6 +66,30 @@ def setup():
sys.exit(1)
print(args)
subprocess.run(args)
setup_binfmt()
def setup_binfmt():
"""Register qemu-user-static binfmt handlers (needed on Arch, etc.)."""
binfmt_conf = """\
:qemu-aarch64:M::\\x7fELF\\x02\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\xb7\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff\\xff:/usr/bin/qemu-aarch64-static:F
:qemu-arm:M::\\x7fELF\\x01\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x28\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff\\xff:/usr/bin/qemu-arm-static:F
:qemu-armeb:M::\\x7fELF\\x01\\x02\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x28:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff:/usr/bin/qemu-armeb-static:F
:qemu-riscv64:M::\\x7fELF\\x02\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\xf3\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff\\xff:/usr/bin/qemu-riscv64-static:F
:qemu-i386:M::\\x7fELF\\x01\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x03\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xfe\\xff\\xff\\xff:/usr/bin/qemu-i386-static:F
"""
conf_path = '/etc/binfmt.d/qemu-static.conf'
print(f"Writing {conf_path}")
try:
with open(conf_path, 'w') as f:
f.write(binfmt_conf)
except PermissionError:
print(f"Permission denied writing {conf_path}; trying with sudo...")
subprocess.run(['sudo', 'tee', conf_path],
input=binfmt_conf.encode(), check=True)
print("Restarting systemd-binfmt...")
subprocess.run(['sudo', 'systemctl', 'restart', 'systemd-binfmt'])
def podman_cleanup():
@@ -92,7 +116,7 @@ def parse_args():
required=False,
action='store_true')
p.add_argument('--git-repo',
required=True,
required=False,
type=str)
return p.parse_args()
@@ -112,6 +136,10 @@ def main():
setup()
sys.exit(0)
if not args.git_repo:
print("error: the following arguments are required: --git-repo")
sys.exit(1)
containerfiles = []
basepath = 'buildtools/containerfiles'
if os.path.exists(f'{basepath}/{args.target}'):
+10
View File
@@ -0,0 +1,10 @@
FROM --platform=amd64 fedora:44 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
+10
View File
@@ -0,0 +1,10 @@
FROM --platform=arm64 fedora:44 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
+7 -3
View File
@@ -1,9 +1,13 @@
ARG BUILD_TIMESTAMP=0
FROM --platform=linux/amd64 ubuntu:latest as build
RUN apt-get update && apt-get install -y makeself
COPY install-build-pkgs /tmp/
RUN /tmp/install-build-pkgs && \
DEBIAN_FRONTEND=noninteractive apt-get -qy --no-install-suggests --no-install-recommends install \
makeself
RUN mkdir -p /tmp/mergerfs/
RUN --mount=type=bind,src=pkgs,dst=/pkgs ls -lhd /pkgs
RUN --mount=type=bind,src=pkgs,dst=/pkgs cp -v /pkg/*static*.gz /tmp/mergerfs/
RUN ls -lhd /pkgs
RUN cp -v /pkgs/*static*.gz /tmp/mergerfs/
RUN mkdir -p /build
RUN makeself /tmp/mergerfs/ /build/mergerfs-installer.run "mergerfs" echo "foo"
FROM scratch
@@ -0,0 +1,10 @@
FROM --platform=amd64 ubuntu:26.04 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
@@ -0,0 +1,10 @@
FROM --platform=arm64 ubuntu:26.04 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
@@ -0,0 +1,10 @@
FROM --platform=armhf ubuntu:26.04 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
@@ -0,0 +1,10 @@
FROM --platform=riscv64 ubuntu:26.04 as build
COPY install-build-pkgs build-mergerfs /tmp/
RUN /tmp/install-build-pkgs
ARG BUILD_TIMESTAMP=0
ARG GIT_REPO
ARG BRANCH
RUN /tmp/build-mergerfs $GIT_REPO $BRANCH
FROM scratch
COPY --from=build /build/ /
+126 -3
View File
@@ -1,39 +1,162 @@
#!/bin/sh
set -e
BUILD_ENV=/tmp/build-env
write_gcc_env()
{
{
printf 'export CC=%s\n' "$1"
printf 'export CXX=%s\n' "$2"
} > "$BUILD_ENV"
}
write_toolset_env()
{
{
printf '. %s\n' "$1"
printf 'export CC=gcc\n'
printf 'export CXX=g++\n'
} > "$BUILD_ENV"
}
latest_apt_gxx()
{
apt-cache --names-only search '^g[+][+]-[0-9]' \
| sed -n 's/^\(g++-[0-9][0-9]*\) .*/\1/p' \
| sort -t- -k2,2V \
| tail -n 1
}
latest_dnf_toolset()
{
dnf -q list --showduplicates 'gcc-toolset-*-gcc-c++' 2>/dev/null \
| sed -n 's/^gcc-toolset-\([0-9][0-9]*\)-gcc-c++\..*/\1/p' \
| sort -V \
| tail -n 1
}
latest_yum_toolset()
{
yum -q list available 'devtoolset-*-gcc-c++' 2>/dev/null \
| sed -n 's/^devtoolset-\([0-9][0-9]*\)-gcc-c++\..*/\1/p' \
| sort -V \
| tail -n 1
}
version_gt()
{
highest="$(printf '%s\n%s\n' "$1" "$2" | sort -V | tail -n 1)"
[ "$highest" = "$1" ] && [ "$1" != "$2" ]
}
check_cxx20()
{
cxx="${CXX:-g++}"
test_src=/tmp/mergerfs-cxx20-check.cpp
test_bin=/tmp/mergerfs-cxx20-check
cat > "$test_src" <<'EOF'
template <typename T>
concept Incrementable = requires(T value) { value + 1; };
static_assert(Incrementable<int>);
int main()
{
return 0;
}
EOF
if ! "$cxx" -std=c++20 "$test_src" -o "$test_bin" >/dev/null 2>&1; then
echo "error: $cxx does not support C++20" >&2
exit 1
fi
rm -f "$test_src" "$test_bin"
}
: > "$BUILD_ENV"
if [ -e /usr/bin/apt-get ]; then
export DEBIAN_FRONTEND=noninteractive
apt-get -qy update
cxx_pkg="$(latest_apt_gxx)"
if [ -n "$cxx_pkg" ]; then
gcc_pkg="gcc-${cxx_pkg#g++-}"
compiler_pkgs="$gcc_pkg $cxx_pkg"
write_gcc_env "$gcc_pkg" "$cxx_pkg"
else
compiler_pkgs="g++"
write_gcc_env gcc g++
fi
apt-get -qy --force-yes --no-install-suggests --no-install-recommends \
install \
ca-certificates \
build-essential \
dpkg-dev \
git \
g++ \
$compiler_pkgs \
debhelper \
lsb-release \
fakeroot
. "$BUILD_ENV"
check_cxx20
elif [ -e /usr/bin/dnf ]; then
dnf -y update
toolset="$(latest_dnf_toolset)"
dnf -y install \
git rpm-build gcc-c++ make which libatomic
default_gxx_major="$(g++ -dumpversion | cut -f1 -d.)"
if [ -n "$toolset" ] && version_gt "$toolset" "$default_gxx_major"; then
dnf -y install \
"gcc-toolset-${toolset}-gcc" \
"gcc-toolset-${toolset}-gcc-c++"
write_toolset_env "/opt/rh/gcc-toolset-${toolset}/enable"
else
write_gcc_env gcc g++
fi
. "$BUILD_ENV"
check_cxx20
elif [ -e /usr/bin/yum ]; then
yum -y update
yum -y install \
git rpm-build gcc-c++ make which
yum -y install centos-release-scl
yum -y install devtoolset-9-gcc\*
yum -y install centos-release-scl || true
yum -y update
toolset="$(latest_yum_toolset)"
default_gxx_major="$(g++ -dumpversion | cut -f1 -d.)"
if [ -n "$toolset" ] && version_gt "$toolset" "$default_gxx_major"; then
yum -y install \
"devtoolset-${toolset}-gcc" \
"devtoolset-${toolset}-gcc-c++"
write_toolset_env "/opt/rh/devtoolset-${toolset}/enable"
else
write_gcc_env gcc g++
fi
. "$BUILD_ENV"
check_cxx20
elif [ -e /usr/bin/zypper ]; then
zypper update -y
zypper install -y \
git rpm-build gcc-c++ make which
write_gcc_env gcc g++
. "$BUILD_ENV"
check_cxx20
elif [ -e /sbin/apk ]; then
apk add \
abuild git gcc g++ make \
linux-headers
write_gcc_env gcc g++
. "$BUILD_ENV"
check_cxx20
elif [ -e /usr/sbin/pkg ]; then
pkg install \
git gmake gcc
elif [ -e /usr/bin/pacman ]; then
pacman -Sy --noconfirm base-devel git fakeroot
write_gcc_env gcc g++
. "$BUILD_ENV"
check_cxx20
fi
+4 -1
View File
@@ -6,8 +6,11 @@ export DH_VERBOSE=1
%:
dh $@
override_dh_auto_build:
$(MAKE) release
override_dh_auto_install:
$(MAKE) \
DESTDIR=$(CURDIR)/debian/mergerfs \
PREFIX=/usr \
install
install
+2
View File
@@ -15,6 +15,8 @@ function or have been replaced. **They should not be used.**
* **hard_remove**: Does nothing.
* **kernel_cache**: Use [cache.files=full](cache.md).
* **negative_entry**: Use [cache.negative-entry](cache.md).
* **remember**: Use `never-forget-nodes`.
* **remember-nodes**: Use `never-forget-nodes`.
* **noforget**: Use `never-forget-nodes`.
* **nonempty**: Does nothing.
* **splice_move**: Does nothing.
+5 -6
View File
@@ -213,12 +213,11 @@ config file.
* **passthrough.max-stack-depth=INT**: Set to `1` another filesystem
can be stacked on mergerfs. Set to `2` to have mergerfs stacked over
another filesystem. (default: 1)
* **remember-nodes=INT**: The number of seconds to keep the internal
representation of a file once the OS tells mergerfs it is no longer
needed. Really only needed for [exporting mergerfs via
NFS](../remote_filesystems.md) (default: 0)
* **never-forget-nodes=BOOL**: Sets `remember-nodes` to
infinity. Formerly `noforget`. (default: false)
* **never-forget-nodes=BOOL**: Keep the internal representation of
a file after the OS tells mergerfs it is no longer needed, until the
node is unlinked, removed, or replaced. Really only needed for
[exporting mergerfs via NFS](../remote_filesystems.md). Formerly
`noforget`. (default: false)
* **allow-idmap=BOOL**: Enables idmap (identity mapping). (default:
true)
* **debug**: Enable debugging information. At the moment limited to a
@@ -149,6 +149,70 @@ It is generally recommended to not use user namespacing / id mapping
given the complication it introduces.
## Does mergerfs work with SELinux relabeling?
FUSE filesystems, including mergerfs, do not support per-inode
SELinux labels (see [Red Hat bug
1272868](https://bugzilla.redhat.com/show_bug.cgi?id=1272868) and the
[upstream fuse-devel
thread](https://sourceforge.net/p/fuse/mailman/fuse-devel/thread/CAN9vWDLGA4%3Dg3peKVWkX7gLRiVUUvJAQx9kiXHgqdeX6eh0MYA%40mail.gmail.com/)).
When a container runtime such as Podman or Docker is asked to relabel
a bind mount with the `:z` or `:Z` mount option, it first probes the
source for `security.selinux` xattr support. On a FUSE path that
probe returns `EOPNOTSUPP` and the runtime skips relabeling entirely.
Files inside the mount keep the `fusefs_t` type assigned by the
kernel. No tree walk is performed and no journal noise is produced at
default verbosity. The runtime logs `Labeling not supported on
"<path>"` only at debug level (for example
`podman --log-level=debug`). The flag is therefore a no-op against a
mergerfs mount; not harmful, but misleading.
Containers can still read and write a mergerfs pool without `:z`
because two independent access checks must succeed and both can be
satisfied without any relabel:
* **SELinux:** on Fedora, RHEL / CentOS Stream, and SUSE the shipped
[container-selinux](https://github.com/containers/container-selinux)
policy already permits the `container_t` process type to manage
files of type `fusefs_t`. See `container_selinux(8)`.
* **DAC:** the in-container UID and GID must still be able to read
and write the files on the underlying branches. With rootless
Podman this is most easily arranged by aligning the container UID
with the host owner of the branches (for example
`--userns=keep-id`, or Quadlet `UserNS=keep-id`).
The recommendation is to omit `:z` and `:Z` on bind mounts whose
source is a mergerfs (or any other FUSE) path, and to keep them on
bind mounts whose source is a regular local filesystem (ext4, btrfs,
xfs, etc.) where relabeling does real work. On distributions whose
container-selinux policy does not already include the `fusefs_t`
allow rule, the targeted fix is to set a single label at mount time
by adding `context="system_u:object_r:container_file_t:s0"` to the
mergerfs mount options (for example in `/etc/fstab`). Files inside
the pool then appear as `container_file_t`, which container-selinux
permits `container_t` to manage everywhere. See [Red Hat's mounting
file systems
guide](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-security-enhanced_linux-working_with_selinux-mounting_file_systems).
This is preferred over the much broader
`--security-opt label=disable`, which removes SELinux confinement
from the container as a whole. The trade-off of `context=` is that
host-side processes touching the mount (rsync, backup tools, web
UIs, cron jobs) see `container_file_t` rather than `fusefs_t` and
may require a small SELinux policy adjustment.
To verify on your own system:
```
podman --log-level=debug run --rm \
-v /path/to/mergerfs:/data:z \
alpine true 2>&1 | grep -iE 'relabel|setfilecon|Labeling'
```
Expected output is a single line of the form
`Labeling not supported on "/path/to/mergerfs"` and nothing in
`ausearch -m AVC,SELINUX_ERR` for the mount.
## Can mergerfs be used with NFS root squash?
If mergerfs is pooling a NFS mount then root squash must be disabled
+25 -1
View File
@@ -2,6 +2,30 @@
## mergerfs
### Advisory file locking not currently supported
mergerfs does not implement POSIX or BSD advisory file locks as
provided by FUSE. Doing so is complex because, as a union filesystem,
it would need to forward lock requests to the underlying branches,
which could result in blocking calls. Locally this is not a major
concern—the kernel still manages locks for applications that all go
through the mergerfs mount. However, if an application locks a file
via the mergerfs mount while another locks the same file directly on
the underlying filesystem, the locks will not interact as one might
expect. That usage pattern is uncommon, so it is generally not a
problem.
The issue becomes more significant if you rely on locks over a remote
filesystem like NFS across multiple hosts, because mergerfs never
actually makes a lock call to the underlying branch. While network
locks are possible, they are generally discouraged, so this may not be
a huge loss either.
That said, there is a strategy that could add lock support if there is
sufficient demand, though it would probably be opt-in for performance
and resource-usage reasons.
### FreeBSD version
* [https://wiki.freebsd.org/FUSEFS](https://wiki.freebsd.org/FUSEFS)
@@ -27,7 +51,7 @@
* ...
#### Host vs Container identity
### Host vs Container identity
While **not** a bug some users have found when using containers that
supplemental groups defined inside the container don't work as
+5 -6
View File
@@ -57,7 +57,7 @@ NFS export settings:
* `softreval`: NFS client will serve up cached data after `retrans`
attempts to revalidate the data. Helps with intermittent network
issues.
* `timeo=150`: Timeout till retrying request.
* `timeo=150`: Timeout till retrying request.
* `retrans=3`: Number of retrying a request.
* `rsize=131072`: Read size.
* `wsize=131072`: Write size.
@@ -76,12 +76,11 @@ but if you run into problems it may be worth trying Samba/SMB.
**mergerfs settings:**
* `never-forget-nodes=true` or `remember-nodes=SECONDS`
* `never-forget-nodes=true`
* `inodecalc=path-hash`
* `lazy-umount-mountpoint=false`
`never-forget-nodes=true` (or `remember-nodes` with a large value such
as `86400` for a full day) is needed because NFS uses the
`never-forget-nodes=true` is needed because NFS uses the
`name_to_handle_at` and `open_by_handle_at` functions which allow a
program to keep a reference to a file without technically having it
open in the typical sense. The problem is that FUSE has no way to know
@@ -89,8 +88,8 @@ that NFS has a handle that it will later use to open the file
again. As a result, it is possible for the kernel to tell mergerfs to
forget about the file node and should NFS ever ask for that node's
details in the future it would have nothing to respond with. Keeping
nodes around forever (or however long is set with `remember-nodes`) is
not ideal but at the moment the only way to manage the situation.
nodes around until they are unlinked, removed, or replaced is not ideal
but at the moment the only way to manage the situation.
`inodecalc=path-hash` is needed because NFS is sensitive to
out-of-band changes. FUSE doesn't care if a file's inode value changes
+1 -1
View File
@@ -12,7 +12,7 @@
depending on [fuse-msg-size](config/fuse-msg-size.md)
* buffers allocated temporarily for reading directories
* FUSE nodes
* never-forget-nodes=true nodes
* never-forget-nodes=true retained nodes, until unlinked or removed
## Management
+23 -1
View File
@@ -135,11 +135,33 @@ docker pull ghcr.io/trapexit/mergerfs:TAG
docker run --device=/dev/fuse --cap-add=SYS_ADMIN -v /mnt/to-merge:/mnt/to-merge:rshared -v /mnt/mergerfs:/mnt/mergerfs:z,shared ghcr.io/trapexit/mergerfs:TAG -f '/mnt/to-merge/*' '/mnt/mergerfs'
```
or with docker compose
```
services:
mergerfs:
image: ghcr.io/trapexit/mergerfs:TAG
container_name: mergerfs
restart: unless-stopped
privileged: true
cap_add:
- SYS_ADMIN
devices:
- /dev/fuse:/dev/fuse
command: -f '/mnt/to-merge/*' '/mnt/mergerfs'
volumes:
- /mnt/to-merge:/mnt/to-merge:rshared
- /mnt/mergerfs:/mnt/mergerfs:z,shared
```
* `--device=/dev/fuse`: Pass in host FUSE device.
* `--cap-add=SYS_ADMIN`: Give proper permissions to mount.
* `-v /mnt/mergerfs:/mnt/mergerfs:z,shared`: `z` and `shared` will
allow the mount to be shared between the container and host and
therefore other containers.
therefore other containers. Note that once mergerfs is mounted
inside the container the resulting FUSE filesystem does not
support SELinux relabeling. See [Does mergerfs work with SELinux
relabeling?](../faq/compatibility_and_integration.md#does-mergerfs-work-with-selinux-relabeling)
* `-v /mnt/to-merge:/mnt/to-merge:rshared`: The mounts from the host
to merge within the container.
+68 -28
View File
@@ -28,8 +28,10 @@
#include "str.hpp"
#include "syslog.hpp"
#include <string>
#include <mutex>
#include <optional>
#include <shared_mutex>
#include <string>
#include <fnmatch.h>
@@ -437,24 +439,44 @@ Branches::from_string(const std::string_view str_)
Branches::Ptr impl;
Branches::Ptr new_impl;
impl = std::atomic_load(&_impl);
new_impl = std::make_shared<Branches::Impl>(&minfreespace);
*new_impl = *impl;
rv = new_impl->from_string(str_);
if(rv < 0)
return rv;
for(;;)
{
{
std::shared_lock<std::shared_mutex> lk(_mutex);
impl = _impl;
}
std::atomic_store(&_impl,new_impl);
*new_impl = *impl;
return 0;
rv = new_impl->from_string(str_);
if(rv < 0)
return rv;
{
std::unique_lock<std::shared_mutex> lk(_mutex);
if(impl != _impl)
continue;
_impl = std::move(new_impl);
}
return 0;
}
}
std::string
Branches::to_string(void) const
{
return std::atomic_load(&_impl)->to_string();
Branches::Ptr impl;
{
std::shared_lock<std::shared_mutex> lk(_mutex);
impl = _impl;
}
return impl->to_string();
}
void
@@ -462,32 +484,50 @@ Branches::find_and_set_mode_ro()
{
Branches::Ptr impl;
Branches::Ptr new_impl;
std::vector<fs::path> ro_paths;
impl = std::atomic_load(&_impl);
new_impl = std::make_shared<Branches::Impl>(&minfreespace);
*new_impl = *impl;
bool changed = false;
for(auto &branch : *new_impl)
for(;;)
{
if(branch.mode != Branch::Mode::RW)
continue;
ro_paths.clear();
if(!fs::is_rofs_but_not_mounted_ro(branch.path))
continue;
{
std::shared_lock<std::shared_mutex> lk(_mutex);
impl = _impl;
}
SysLog::warning("branch `{}` found to be readonly - setting its mode=RO",
branch.path.string());
*new_impl = *impl;
branch.mode = Branch::Mode::RO;
changed = true;
for(auto &branch : *new_impl)
{
if(branch.mode != Branch::Mode::RW)
continue;
if(!fs::is_rofs_but_not_mounted_ro(branch.path))
continue;
branch.mode = Branch::Mode::RO;
ro_paths.emplace_back(branch.path);
}
if(ro_paths.empty())
return;
{
std::unique_lock<std::shared_mutex> lk(_mutex);
if(impl != _impl)
continue;
_impl = std::move(new_impl);
}
for(const auto &path : ro_paths)
SysLog::warning("branch `{}` found to be readonly - setting its mode=RO",
path.string());
return;
}
if(not changed)
return;
std::atomic_store(&_impl,new_impl);
}
SrcMounts::SrcMounts(Branches &b_)
+14 -4
View File
@@ -23,9 +23,9 @@
#include "strvec.hpp"
#include "tofrom_string.hpp"
#include <atomic>
#include <cstdint>
#include <memory>
#include <shared_mutex>
#include <string>
#include <vector>
@@ -69,7 +69,8 @@ public:
u64 minfreespace = MINFREESPACE_DEFAULT;
private:
mutable std::shared_ptr<Impl> _impl;
Branches::Ptr _impl;
mutable std::shared_mutex _mutex;
public:
Branches()
@@ -82,8 +83,17 @@ public:
std::string to_string(void) const final;
public:
operator Ptr() const { return std::atomic_load(&_impl); }
Ptr operator->() const { return std::atomic_load(&_impl); }
operator Ptr() const
{
std::shared_lock<std::shared_mutex> lk(_mutex);
return _impl;
}
Ptr operator->() const
{
std::shared_lock<std::shared_mutex> lk(_mutex);
return _impl;
}
public:
void find_and_set_mode_ro();
+12 -11
View File
@@ -135,10 +135,6 @@ Config::Config()
_max_background(fuse_cfg.max_background),
_mount(mountpoint),
_mountpoint(mountpoint),
_never_forget_nodes(),
_noforget(),
_remember(fuse_cfg.remember_nodes),
_remember_nodes(fuse_cfg.remember_nodes),
_srcmounts(branches),
_threads(fuse_cfg.read_thread_count),
_uid(fuse_cfg.uid),
@@ -147,13 +143,22 @@ Config::Config()
_initialized(false)
{
allow_idmap.ro =
_congestion_threshold.ro =
_max_background.ro =
_mount.ro =
_mountpoint.ro =
_never_forget_nodes.ro =
_noforget.ro =
_remember.ro =
_remember_nodes.ro =
_srcmounts.ro =
_version.ro =
allow_idmap.ro =
async_read.ro =
branches_mount_timeout.ro =
branches_mount_timeout_fail.ro =
cache_symlinks.ro =
cache_writeback.ro =
_congestion_threshold.ro =
direct_io_allow_mmap.ro =
export_support.ro =
fsname.ro =
@@ -161,9 +166,6 @@ Config::Config()
handle_killpriv.ro =
handle_killpriv_v2.ro =
kernel_permissions_check.ro =
_max_background.ro =
_mount.ro =
_mountpoint.ro =
nullrw.ro =
pid.ro =
pin_threads.ro =
@@ -172,8 +174,6 @@ Config::Config()
process_thread_queue_depth.ro =
read_thread_count.ro =
scheduling_priority.ro =
_srcmounts.ro =
_version.ro =
true;
_congestion_threshold.display =
_gid.display =
@@ -181,6 +181,7 @@ Config::Config()
_mount.display =
_noforget.display =
_remember.display =
_remember_nodes.display =
_srcmounts.display =
_threads.display =
_uid.display =
+2 -2
View File
@@ -176,8 +176,8 @@ private:
TFSRef<fs::path> _mountpoint;
CfgNoforget _never_forget_nodes;
CfgNoforget _noforget;
TFSRef<s64> _remember;
TFSRef<s64> _remember_nodes;
CfgNoforget _remember;
CfgNoforget _remember_nodes;
SrcMounts _srcmounts;
TFSRef<int> _threads;
TFSRef<s64> _uid;
+15 -9
View File
@@ -32,18 +32,24 @@ public:
{
int rv;
bool b = false;
s64 i = 0;
rv = str::from(s_,&b);
if((b == true) || s_.empty())
if(s_.empty())
{
fuse_cfg.remember_nodes = -1;
return 0;
b = true;
}
else if((rv = str::from(s_,&i)) == 0)
{
b = (i != 0);
}
else
{
rv = str::from(s_,&b);
if(rv)
return rv;
}
if(rv)
return rv;
fuse_cfg.remember_nodes = 0;
fuse_cfg.remember_nodes = b;
return 0;
}
@@ -51,7 +57,7 @@ public:
std::string
to_string() const
{
if(fuse_cfg.remember_nodes == -1)
if(fuse_cfg.remember_nodes)
return "true";
return "false";
}
+12 -3
View File
@@ -48,10 +48,10 @@
#include "fh.hpp"
#include "fs_path.hpp"
#include "mutex.hpp"
#include "base_types.h"
#include <shared_mutex>
class FileInfo : public FH
{
@@ -97,7 +97,16 @@ public:
int fd;
Branch branch;
u32 direct_io:1;
Mutex mutex;
// Serializes the fd state across concurrent writes on the same open
// file. Concurrent writes happen with:
// 1) writeback-cache + page-cache mode
// 2) parallel_direct_writes + direct_io=true
//
// Common case (plain pwrite on stable fd) only needs a shared lock
// since the kernel already serializes fd mutations. The danger is
// the fs::dup2() rebinding when moveonenospc activates. Writers
// take shared lock; move takes unique.
std::shared_mutex mutex;
};
inline
-40
View File
@@ -1,40 +0,0 @@
/*
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#pragma once
#include "to_neg_errno.hpp"
#include <sys/file.h>
namespace fs
{
static
inline
int
flock(const int fd_,
const int operation_)
{
int rv;
rv = ::flock(fd_,operation_);
return ::to_neg_errno(rv);
}
}
+24 -16
View File
@@ -21,8 +21,8 @@
#include "fs_statvfs.hpp"
#include "statvfs_util.hpp"
#include "mutex.hpp"
#include <mutex>
#include <shared_mutex>
#include <string>
#include <unordered_map>
@@ -38,9 +38,11 @@ struct Element
typedef std::unordered_map<std::string,Element> statvfs_cache;
static u64 g_timeout = 0;
static statvfs_cache g_cache;
static Mutex g_cache_lock;
// Cache is read heavy. Use a shared mutex so concurrent reads can
// work in parallel and only inserts or refreshes take a unique lock.
static std::shared_mutex g_cache_mutex;
static u64 g_timeout = 0;
static statvfs_cache g_cache;
static
@@ -70,29 +72,35 @@ int
fs::statvfs_cache(const std::string &path_,
struct statvfs *st_)
{
int rv;
Element *e;
u64 now;
if(g_timeout == 0)
return fs::statvfs(path_,st_);
rv = 0;
now = ::_get_time();
mutex_lock(g_cache_lock);
{
std::shared_lock<std::shared_mutex> lk(g_cache_mutex);
e = &g_cache[path_];
auto it = g_cache.find(path_);
if((it != g_cache.end()) && ((now - it->second.time) <= g_timeout))
{
*st_ = it->second.st;
return 0;
}
}
if((now - e->time) > g_timeout)
int rv = 0;
std::unique_lock<std::shared_mutex> lk(g_cache_mutex);
Element &e = g_cache[path_];
if((now - e.time) > g_timeout)
{
e->time = now;
rv = fs::statvfs(path_,&e->st);
e.time = now;
rv = fs::statvfs(path_,&e.st);
}
*st_ = e->st;
mutex_unlock(g_cache_lock);
*st_ = e.st;
return rv;
}
+1 -1
View File
@@ -334,7 +334,7 @@ _create(const fuse_req_ctx_t *ctx_,
delete fi;
ffi_->fh = 0;
SysLog::crit(msg);
SysLog::crit("{}",msg);
fmt::println(stderr,"{}",msg);
return -EIO;
-60
View File
@@ -1,60 +0,0 @@
/*
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include "fuse_flock.hpp"
#include "errno.hpp"
#include "fileinfo.hpp"
#include "fs_flock.hpp"
#include "state.hpp"
#include "fuse.h"
static
int
_flock(const int fd_,
const int operation_)
{
int rv;
rv = fs::flock(fd_,operation_);
return rv;
}
// In cases where the node was opened but unlinked there will be no
// path and no guarentee of a `fh`. It could always do the lookup but
// why bother if the kernel has provided it to us? The reason the
// function doesn't need to run within the visit lambda is because
// since the request is outstanding the kernel won't be releasing the
// node and therefore the entry will be valid over the lifetime of
// this function.
int
FUSE::flock(const fuse_req_ctx_t *ctx_,
const fuse_file_info_t *ffi_,
int op_)
{
FileInfo *fi;
fi = state.get_fi(ctx_,ffi_->fh);
if(not fi)
return -EBADF;
return ::_flock(fi->fd,op_);
}
-30
View File
@@ -1,30 +0,0 @@
/*
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#pragma once
#include "fuse.h"
namespace FUSE
{
int
flock(const fuse_req_ctx_t *ctx,
const fuse_file_info_t *ffi,
int op);
}
-2
View File
@@ -209,8 +209,6 @@ FUSE::init(fuse_conn_info_t *conn_)
::_want_if_capable(conn_,FUSE_CAP_ALLOW_IDMAP,&cfg.allow_idmap);
// ::_want_if_capable(conn_,FUSE_CAP_READDIR_PLUS_AUTO);
::_want_if_capable_max_pages(conn_,cfg);
conn_->want &= ~FUSE_CAP_POSIX_LOCKS;
conn_->want &= ~FUSE_CAP_FLOCK_LOCKS;
::_spawn_thread_to_set_readahead();
-33
View File
@@ -1,33 +0,0 @@
/*
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include "fuse_lock.hpp"
#include "errno.hpp"
#include "fuse.h"
int
FUSE::lock(const fuse_req_ctx_t *ctx,
const fuse_file_info_t *ffi,
int cmd,
struct flock *flock)
{
return -ENOSYS;
}
-30
View File
@@ -1,30 +0,0 @@
/*
ISC License
Copyright (c) 2026, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#pragma once
#include "fuse.h"
namespace FUSE
{
int
lock(const fuse_req_ctx_t *ctx,
const fuse_file_info_t *ffi,
int cmd,
struct flock *flock);
}
+10 -1
View File
@@ -367,10 +367,19 @@ _open(const fuse_req_ctx_t *ctx_,
// The ref increment keeps fuse_release.cpp from erasing and
// freeing the canonical entry while we duplicate its fd.
//
// memory_order_relaxed is sufficient for the increment. The
// concurrent_flat_map bucket lock held inside visit() already
// provides the happens-before relationship with other mutators
// of the same key (the release() erase_if() callback and other
// open() visitors.) The lock's release/acquire semantics
// publish the ref_count update to subsequent observers of the
// same entry, so we only need atomicity from the fetch_add
// itself.
of.visit(ctx_->nodeid,
[&](auto &v_)
{
v_.second.ref_count++;
v_.second.ref_count.fetch_add(1,std::memory_order_relaxed);
fi = v_.second.fi;
backing_id = v_.second.backing_id;
});
+21 -15
View File
@@ -48,30 +48,29 @@ FUSE::ReadDir::ReadDir(const std::string_view s_)
std::string
FUSE::ReadDir::to_string() const
{
std::shared_ptr<std::string> str;
std::shared_lock<std::shared_mutex> lk(_mutex);
str = std::atomic_load(&_str);
return (*str);
return _str;
}
int
FUSE::ReadDir::from_string(const std::string_view str_)
{
std::shared_ptr<std::string> tmp_str;
std::shared_ptr<FUSE::ReadDirBase> tmp_readdir;
tmp_readdir = FUSE::ReadDirFactory::make(str_);
if(!tmp_readdir)
return -EINVAL;
tmp_str = std::make_shared<std::string>(str_);
std::atomic_store(&_str,tmp_str);
{
std::unique_lock<std::shared_mutex> lk(_mutex);
if(_initialized == false)
return 0;
_str = str_;
if(_initialized == false)
return 0;
std::atomic_store(&_impl,tmp_readdir);
_impl = std::move(tmp_readdir);
}
return 0;
}
@@ -114,7 +113,10 @@ FUSE::ReadDir::operator()(const fuse_req_ctx_t *ctx_,
int rv;
std::shared_ptr<FUSE::ReadDirBase> readdir;
readdir = std::atomic_load(&_impl);
{
std::shared_lock<std::shared_mutex> lk(_mutex);
readdir = _impl;
}
if(!readdir)
fatal::abort("readdir impl is null");
@@ -128,10 +130,14 @@ FUSE::ReadDir::operator()(const fuse_req_ctx_t *ctx_,
void
FUSE::ReadDir::initialize()
{
std::shared_ptr<std::string> str;
std::string str;
str = std::atomic_load(&_str);
_initialized = true;
{
std::unique_lock<std::shared_mutex> lk(_mutex);
from_string(*str);
str = _str;
_initialized = true;
}
from_string(str);
}
+4 -3
View File
@@ -22,9 +22,9 @@
#include "tofrom_string.hpp"
#include "fuse_readdir_base.hpp"
#include <atomic>
#include <memory>
#include <mutex>
#include <shared_mutex>
namespace FUSE
@@ -59,8 +59,9 @@ namespace FUSE
void initialize();
private:
std::atomic<bool> _initialized = false;
std::shared_ptr<std::string> _str;
bool _initialized = false;
mutable std::shared_mutex _mutex;
std::string _str;
std::shared_ptr<FUSE::ReadDirBase> _impl;
};
}
+6
View File
@@ -43,6 +43,12 @@ FUSE::ReadDirCOR::~ReadDirCOR()
#include "fuse_readdir_cor_readdir.icpp"
#endif
// Uses a per-call mutex. Lifetime is one readdir operation. It
// serializes the dirent-merge step (dedup'ing of names + append to
// dirents list) across at-most-branches-worth of parallel worker
// tasks. Per-branch opendir + getdents64 run fully outside the
// critical section, so the lock holds only long enough to merge each
// batch.
static
inline
int
+12 -2
View File
@@ -81,11 +81,21 @@ FUSE::release(cu64 nodeid_,
FileInfo *canonical_fi = nullptr;
FileInfo *fh_fi_to_free = nullptr;
// memory_order_acq_rel on the decrement gives us the canonical
// refcount-publication pattern:
// - release: prior writes to the OpenFile by this thread are
// visible to whichever thread observes the final 0.
// - acquire: the thread that drops the last ref acquires every
// prior ref holder's writes before tearing down the entry.
// The concurrent_flat_map bucket lock already serializes concurrent
// visitors of this key so the acq_rel is good for any observer that
// might examine ref_count outside a visit().
of.erase_if(nodeid_,
[&](auto &v_)
{
v_.second.ref_count--;
if(v_.second.ref_count > 0)
const int prev =
v_.second.ref_count.fetch_sub(1,std::memory_order_acq_rel);
if(prev > 1)
{
if(fi_ != v_.second.fi)
fh_fi_to_free = fi_;
+50 -46
View File
@@ -33,6 +33,9 @@
#include "fuse.h"
#include <mutex>
#include <shared_mutex>
static
bool
@@ -117,47 +120,11 @@ _move_and_pwriten(char const *buf_,
// N bytes written (short writes included)
// -errno on error
// See libfuse/include/fuse.h for more details
static
int
_write_direct_io(const char *buf_,
const size_t count_,
const off_t offset_,
FileInfo *fi_)
{
ssize_t rv;
rv = fs::pwrite(fi_->fd,buf_,count_,offset_);
if(::_out_of_space(rv))
rv = ::_move_and_pwrite(buf_,count_,offset_,fi_,rv);
return rv;
}
// When not in direct_io mode write's return value is more complex.
// 0 or less than `count` on EOF
// `count` on non-errors
// -errno on error
// See libfuse/include/fuse.h for more details
static
int
_write_cached(const char *buf_,
const size_t count_,
const off_t offset_,
FileInfo *fi_)
{
int err;
ssize_t rv;
rv = fs::pwriten(fi_->fd,buf_,count_,offset_,&err);
if(err == 0)
return rv;
if(err && !::_out_of_space(err))
return err;
rv = ::_move_and_pwriten(buf_,count_,offset_,fi_,err,rv);
return rv;
}
static
int
@@ -175,19 +142,56 @@ _write(const fuse_req_ctx_t *ctx_,
// Concurrent writes can only happen if:
// 1) writeback-cache is enabled and using page caching
// 2) parallel_direct_writes is enabled and file has `direct_io=true`
// Will look into selectively locking in the future
// A reader/writer lock would probably be the best option given
// the expense of the write itself in comparison. Alternatively,
// could change the move file behavior to use a known target file
// and have threads use O_EXCL and back off and wait for the
// transfer to complete before retrying.
mutex_lockguard(fi->mutex);
// 2) parallel_direct_writes is enabled and file has
// `direct_io=true`
ssize_t rv;
if(fi->direct_io)
return ::_write_direct_io(buf_,count_,offset_,fi);
{
{
std::shared_lock<std::shared_mutex> slk(fi->mutex);
rv = fs::pwrite(fi->fd,buf_,count_,offset_);
}
return ::_write_cached(buf_,count_,offset_,fi);
if(not ::_out_of_space(rv))
return rv;
std::unique_lock<std::shared_mutex> ulk(fi->mutex);
// Re-check under exclusive lock: another writer may have
// already moved the file and retired. If so the pwrite should
// now succeed without our own move call.
rv = fs::pwrite(fi->fd,buf_,count_,offset_);
if(::_out_of_space(rv))
rv = ::_move_and_pwrite(buf_,count_,offset_,fi,rv);
return rv;
}
else
{
int err;
ssize_t written;
{
std::shared_lock<std::shared_mutex> slk(fi->mutex);
written = fs::pwriten(fi->fd,buf_,count_,offset_,&err);
}
if(err == 0)
return written;
if(err && not ::_out_of_space(err))
return err;
std::unique_lock<std::shared_mutex> ulk(fi->mutex);
// Re-check under exclusive lock as another move may have
// already run.
written = fs::pwriten(fi->fd,buf_,count_,offset_,&err);
if(err == 0)
return written;
if(err && not ::_out_of_space(err))
return err;
return ::_move_and_pwriten(buf_,count_,offset_,fi,err,written);
}
}
int
+8 -12
View File
@@ -32,8 +32,12 @@ enum
IOPRIO_WHO_PROCESS = 1
};
namespace ioprio
{
std::atomic<bool> _enabled{false};
}
thread_local int ioprio::SetFrom::thread_prio = -1;
bool _enabled = false;
int
ioprio::get(const int who_)
@@ -69,22 +73,14 @@ ioprio::set(const int who_,
void
ioprio::enable(const bool enable_)
{
_enabled = enable_;
_enabled.store(enable_,std::memory_order_relaxed);
}
bool
ioprio::enabled()
{
return _enabled;
}
ioprio::SetFrom::SetFrom(const pid_t pid_)
void
ioprio::SetFrom::_slow_apply(const pid_t pid_)
{
int client_prio;
if(!_enabled)
return;
client_prio = ioprio::get(pid_);
if(client_prio < 0)
return;
+25 -2
View File
@@ -18,13 +18,27 @@
#pragma once
#include <atomic>
#include <sys/types.h>
namespace ioprio
{
// Exposed so the disabled-path check in SetFrom can inline into
// every fuse_read/fuse_write as an atomic load + branch, with no
// function overhead.
extern std::atomic<bool> _enabled;
[[gnu::always_inline]]
inline
bool
enabled()
{
return _enabled.load(std::memory_order_relaxed);
}
void enable(const bool);
bool enabled();
int get(const int who);
int set(const int who, const int ioprio);
@@ -33,6 +47,15 @@ namespace ioprio
{
static thread_local int thread_prio;
SetFrom(const pid_t pid);
[[gnu::always_inline]]
inline
SetFrom(const pid_t pid_)
{
if(ioprio::enabled())
_slow_apply(pid_);
}
private:
void _slow_apply(const pid_t);
};
};
+1 -5
View File
@@ -46,7 +46,6 @@
#include "fuse_fchmod.hpp"
#include "fuse_fchown.hpp"
#include "fuse_fgetattr.hpp"
#include "fuse_flock.hpp"
#include "fuse_flush.hpp"
#include "fuse_fsync.hpp"
#include "fuse_fsyncdir.hpp"
@@ -58,7 +57,6 @@
#include "fuse_ioctl.hpp"
#include "fuse_link.hpp"
#include "fuse_listxattr.hpp"
#include "fuse_lock.hpp"
#include "fuse_mkdir.hpp"
#include "fuse_mknod.hpp"
#include "fuse_open.hpp"
@@ -111,7 +109,6 @@ _get_fuse_operations(struct fuse_operations &ops_,
ops_.fchmod = FUSE::fchmod;
ops_.fchown = FUSE::fchown;
ops_.fgetattr = FUSE::fgetattr;
ops_.flock = FUSE::flock;
ops_.flush = FUSE::flush;
ops_.fsync = FUSE::fsync;
ops_.fsyncdir = FUSE::fsyncdir;
@@ -123,7 +120,6 @@ _get_fuse_operations(struct fuse_operations &ops_,
ops_.ioctl = FUSE::ioctl;
ops_.link = FUSE::link;
ops_.listxattr = FUSE::listxattr;
ops_.lock = FUSE::lock;
ops_.mkdir = FUSE::mkdir;
ops_.mknod = FUSE::mknod;
ops_.open = FUSE::open;
@@ -305,7 +301,7 @@ _warn_if_not_root()
"mergerfs is not running as root"
" and may not work correctly";
fmt::println(stderr,"* WARNING: {}",s);
SysLog::warning(s);
SysLog::warning("{}",s);
}
int
+5 -1
View File
@@ -103,8 +103,12 @@ public:
{
}
// Move constructor runs during initial insertion into the
// concurrent_flat_map before the new entry is reachable by the
// other threads. There is no concurrent observer of the moved
// value at this point so relaxed load is sufficient.
OpenFile(OpenFile &&o_) noexcept
: ref_count(o_.ref_count.load()),
: ref_count(o_.ref_count.load(std::memory_order_relaxed)),
backing_id(o_.backing_id),
fi(o_.fi)
{
+2 -2
View File
@@ -100,8 +100,8 @@ Legend:
| Option | Integration tests | Unit tests |
|---|---|---|
| `lazy-umount-mountpoint` | Live-upgrade remount scenario with stale previous mount | Startup control flow and error handling |
| `remember-nodes` | NFS-like lookup stability over configured retention interval | Node cache retention/expiry logic |
| `noforget` | No node expiry over extended period | Infinite retention mode mapping |
| `remember-nodes` | NFS-like lookup stability until unlink/remove/rename replacement | Boolean and numeric compatibility parsing |
| `noforget` | Retained nodes released on unlink/remove/rename replacement | Alias mapping to retained-node mode |
| `allow-idmap` | UID/GID mapping behavior in idmapped mounts | Init flag propagation and compatibility checks |
| `debug` | FUSE trace generation enabled/disabled | Debug option parse and logger toggling |
| `log.file` | Trace output to stderr vs file path | Path parser and file sink setup |
+7 -1
View File
@@ -6,10 +6,13 @@ and compare behavior against equivalent native syscalls on a local filesystem.
The suite is "soft POSIX": it focuses on practical parity for return values,
`errno`, and key side effects, including common error paths.
Each test mounts its own ephemeral mergerfs instance with random temporary
branches and cleans everything up automatically.
Run with:
```bash
python3 tests/run-tests /mnt/mergerfs/
python3 tests/run-tests
```
## Coverage Matrix
@@ -69,3 +72,6 @@ Across tests, parity checks include combinations of:
- Tests that cannot exercise a behavior in the current environment may exit 77
and are reported as `SKIP` by `tests/run-tests`.
- The xattr test uses a `user.*` key and assumes user xattrs are enabled.
- Tests mount their own ephemeral mergerfs instances using the `mergerfs_mount`
context manager in `posix_parity.py`. Random directories are created and
cleaned up automatically.
+50 -54
View File
@@ -2,80 +2,76 @@
import os
import sys
import tempfile
from posix_parity import cleanup_dir
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_branches
from posix_parity import mergerfs_get_option
from posix_parity import mergerfs_mount
from posix_parity import mergerfs_set_option
from posix_parity import temp_dir
from posix_parity import touch
from posix_parity import mergerfs_get_option
from posix_parity import mergerfs_set_option
from posix_parity import mergerfs_branches
def main():
if len(sys.argv) != 2:
print("usage: TEST_api_xattr <mountpoint>", file=sys.stderr)
return 1
mount = sys.argv[1]
try:
ctrl = join(mount, ".mergerfs")
if not os.path.exists(ctrl):
return fail(".mergerfs control file not found")
with mergerfs_mount() as (mount, branches):
ctrl = join(mount, ".mergerfs")
if not os.path.exists(ctrl):
return fail(".mergerfs control file not found")
branches = mergerfs_branches(mount)
if not branches:
return fail("no branches found")
merge_base = temp_dir(mount)
try:
merge_file = join(merge_base, "api_test")
touch(merge_file, b"test")
if not branches:
return fail("no branches found")
merge_base = temp_dir(mount)
try:
fullpath = os.getxattr(merge_file, "user.mergerfs.fullpath")
fullpath_str = fullpath.decode("utf-8", errors="surrogateescape")
if not fullpath_str:
return fail("fullpath xattr is empty")
if not os.path.exists(fullpath_str):
return fail(f"fullpath '{fullpath_str}' does not exist on disk")
except OSError as exc:
return fail(f"getxattr fullpath: errno={exc.errno}")
merge_file = join(merge_base, "api_test")
touch(merge_file, b"test")
try:
relpath = os.getxattr(merge_file, "user.mergerfs.relpath")
relpath_str = relpath.decode("utf-8", errors="surrogateescape")
if not relpath_str:
return fail("relpath xattr is empty")
except OSError as exc:
return fail(f"getxattr relpath: errno={exc.errno}")
try:
fullpath = os.getxattr(merge_file, "user.mergerfs.fullpath")
fullpath_str = fullpath.decode("utf-8", errors="surrogateescape")
if not fullpath_str:
return fail("fullpath xattr is empty")
if not os.path.exists(fullpath_str):
return fail(f"fullpath '{fullpath_str}' does not exist on disk")
except OSError as exc:
return fail(f"getxattr fullpath: errno={exc.errno}")
try:
basepath = os.getxattr(merge_file, "user.mergerfs.basepath")
basepath_str = basepath.decode("utf-8", errors="surrogateescape")
if not basepath_str:
return fail("basepath xattr is empty")
except OSError as exc:
return fail(f"getxattr basepath: errno={exc.errno}")
try:
relpath = os.getxattr(merge_file, "user.mergerfs.relpath")
relpath_str = relpath.decode("utf-8", errors="surrogateescape")
if not relpath_str:
return fail("relpath xattr is empty")
except OSError as exc:
return fail(f"getxattr relpath: errno={exc.errno}")
try:
allpaths = os.getxattr(merge_file, "user.mergerfs.allpaths")
allpaths_str = allpaths.decode("utf-8", errors="surrogateescape")
if not allpaths_str:
return fail("allpaths xattr is empty")
except OSError as exc:
return fail(f"getxattr allpaths: errno={exc.errno}")
try:
basepath = os.getxattr(merge_file, "user.mergerfs.basepath")
basepath_str = basepath.decode("utf-8", errors="surrogateescape")
if not basepath_str:
return fail("basepath xattr is empty")
except OSError as exc:
return fail(f"getxattr basepath: errno={exc.errno}")
return 0
finally:
cleanup_dir(merge_base)
try:
allpaths = os.getxattr(merge_file, "user.mergerfs.allpaths")
allpaths_str = allpaths.decode("utf-8", errors="surrogateescape")
if not allpaths_str:
return fail("allpaths xattr is empty")
except OSError as exc:
return fail(f"getxattr allpaths: errno={exc.errno}")
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
except Exception as exc:
return fail(str(exc))
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+82 -85
View File
@@ -10,6 +10,7 @@ from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_branches
from posix_parity import mergerfs_get_option
from posix_parity import mergerfs_mount
from posix_parity import mergerfs_set_option
from posix_parity import touch
@@ -56,96 +57,92 @@ def setup_paths(branches, mount, rel_base, name):
def main():
if len(sys.argv) != 2:
print("usage: TEST_cfg_link_rename_exdev <mountpoint>", file=sys.stderr)
return 1
mount = sys.argv[1]
rel_base = f".mergerfs_exdev_test_{os.getpid()}_{uuid.uuid4().hex}"
branches = mergerfs_branches(mount)
if len(branches) < 2:
print("EXDEV fallback test requires at least 2 branches", end="")
return SKIP
branches = branches[:2]
if os.stat(branches[0]).st_dev == os.stat(branches[1]).st_dev:
print("EXDEV fallback test requires branches on different devices", end="")
return SKIP
try:
try:
orig_create = mergerfs_get_option(mount, "category.create")
orig_link = mergerfs_get_option(mount, "link-exdev")
orig_rename = mergerfs_get_option(mount, "rename-exdev")
except (PermissionError, FileNotFoundError, OSError):
print("EXDEV fallback runtime options unavailable", end="")
return SKIP
try:
# Maximize chance of EXDEV path by path-preserving create policy.
mergerfs_set_option(mount, "category.create", "epmfs")
# link-exdev passthrough => EXDEV when cross-device path preserving applies.
src, dst, src_base = setup_paths(branches, mount, rel_base, "link_pass")
mergerfs_set_option(mount, "link-exdev", "passthrough")
err = get_errno(lambda: os.link(src, dst + ".link.pass"))
if err != errno.EXDEV:
print(f"link-exdev passthrough did not exercise EXDEV, errno={err}", end="")
with mergerfs_mount(num_branches=2) as (mount, branches):
rel_base = f".mergerfs_exdev_test_{os.getpid()}_{uuid.uuid4().hex}"
if len(branches) < 2:
print("EXDEV fallback test requires at least 2 branches", end="")
return SKIP
branches = branches[:2]
if os.stat(branches[0]).st_dev == os.stat(branches[1]).st_dev:
print("EXDEV fallback test requires branches on different devices", end="")
return SKIP
for mode in ("rel-symlink", "abs-base-symlink", "abs-pool-symlink"):
src, dst, src_base = setup_paths(branches, mount, rel_base, f"link_{mode}")
mergerfs_set_option(mount, "link-exdev", mode)
link_path = dst + ".link"
err = get_errno(lambda: os.link(src, link_path))
if err != 0:
return fail(f"link-exdev={mode} returned errno={err}")
err_msg = require_symlink(link_path, f"link-exdev={mode}")
if err_msg:
return fail(err_msg)
resolved = symlink_resolves(link_path)
if mode == "abs-base-symlink" and resolved != os.path.realpath(src_base):
return fail(f"link-exdev={mode}: target {resolved!r} != {os.path.realpath(src_base)!r}")
if mode != "abs-base-symlink" and resolved != os.path.realpath(src):
return fail(f"link-exdev={mode}: target {resolved!r} != {os.path.realpath(src)!r}")
# rename-exdev passthrough => EXDEV in cross-device path preserving case.
src_ren, dst_ren, _ = setup_paths(branches, mount, rel_base, "rename_pass")
mergerfs_set_option(mount, "rename-exdev", "passthrough")
err = get_errno(lambda: os.rename(src_ren, dst_ren + ".ren.pass"))
if err != errno.EXDEV:
print(f"rename-exdev passthrough did not exercise EXDEV, errno={err}", end="")
return SKIP
for mode in ("rel-symlink", "abs-symlink"):
src_ren, dst_ren, _ = setup_paths(branches, mount, rel_base, f"rename_{mode}")
mergerfs_set_option(mount, "rename-exdev", mode)
ren_path = dst_ren + ".ren"
err = get_errno(lambda: os.rename(src_ren, ren_path))
if err != 0:
return fail(f"rename-exdev={mode} returned errno={err}")
err_msg = require_symlink(ren_path, f"rename-exdev={mode}")
if err_msg:
return fail(err_msg)
target = symlink_resolves(ren_path)
if ".mergerfs_rename_exdev" not in target:
return fail(f"rename-exdev={mode}: target not moved under .mergerfs_rename_exdev: {target!r}")
with open(ren_path, "rb") as fp:
if fp.read() != b"src":
return fail(f"rename-exdev={mode}: symlink target content mismatch")
except (PermissionError, FileNotFoundError, OSError):
return SKIP
finally:
try:
mergerfs_set_option(mount, "category.create", orig_create)
mergerfs_set_option(mount, "link-exdev", orig_link)
mergerfs_set_option(mount, "rename-exdev", orig_rename)
except OSError:
pass
try:
orig_create = mergerfs_get_option(mount, "category.create")
orig_link = mergerfs_get_option(mount, "link-exdev")
orig_rename = mergerfs_get_option(mount, "rename-exdev")
except (PermissionError, FileNotFoundError, OSError):
print("EXDEV fallback runtime options unavailable", end="")
return SKIP
return 0
finally:
for branch in branches:
cleanup_dir(join(branch, rel_base))
try:
mergerfs_set_option(mount, "category.create", "epmfs")
src, dst, src_base = setup_paths(branches, mount, rel_base, "link_pass")
mergerfs_set_option(mount, "link-exdev", "passthrough")
err = get_errno(lambda: os.link(src, dst + ".link.pass"))
if err != errno.EXDEV:
print(f"link-exdev passthrough did not exercise EXDEV, errno={err}", end="")
return SKIP
for mode in ("rel-symlink", "abs-base-symlink", "abs-pool-symlink"):
src, dst, src_base = setup_paths(branches, mount, rel_base, f"link_{mode}")
mergerfs_set_option(mount, "link-exdev", mode)
link_path = dst + ".link"
err = get_errno(lambda: os.link(src, link_path))
if err != 0:
return fail(f"link-exdev={mode} returned errno={err}")
err_msg = require_symlink(link_path, f"link-exdev={mode}")
if err_msg:
return fail(err_msg)
resolved = symlink_resolves(link_path)
if mode == "abs-base-symlink" and resolved != os.path.realpath(src_base):
return fail(f"link-exdev={mode}: target {resolved!r} != {os.path.realpath(src_base)!r}")
if mode != "abs-base-symlink" and resolved != os.path.realpath(src):
return fail(f"link-exdev={mode}: target {resolved!r} != {os.path.realpath(src)!r}")
src_ren, dst_ren, _ = setup_paths(branches, mount, rel_base, "rename_pass")
mergerfs_set_option(mount, "rename-exdev", "passthrough")
err = get_errno(lambda: os.rename(src_ren, dst_ren + ".ren.pass"))
if err != errno.EXDEV:
print(f"rename-exdev passthrough did not exercise EXDEV, errno={err}", end="")
return SKIP
for mode in ("rel-symlink", "abs-symlink"):
src_ren, dst_ren, _ = setup_paths(branches, mount, rel_base, f"rename_{mode}")
mergerfs_set_option(mount, "rename-exdev", mode)
ren_path = dst_ren + ".ren"
err = get_errno(lambda: os.rename(src_ren, ren_path))
if err != 0:
return fail(f"rename-exdev={mode} returned errno={err}")
err_msg = require_symlink(ren_path, f"rename-exdev={mode}")
if err_msg:
return fail(err_msg)
target = symlink_resolves(ren_path)
if ".mergerfs_rename_exdev" not in target:
return fail(f"rename-exdev={mode}: target not moved under .mergerfs_rename_exdev: {target!r}")
with open(ren_path, "rb") as fp:
if fp.read() != b"src":
return fail(f"rename-exdev={mode}: symlink target content mismatch")
except (PermissionError, FileNotFoundError, OSError):
return SKIP
finally:
try:
mergerfs_set_option(mount, "category.create", orig_create)
mergerfs_set_option(mount, "link-exdev", orig_link)
mergerfs_set_option(mount, "rename-exdev", orig_rename)
except OSError:
pass
return 0
finally:
for branch in branches:
cleanup_dir(join(branch, rel_base))
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+41 -42
View File
@@ -5,60 +5,59 @@ import sys
from posix_parity import fail
from posix_parity import mergerfs_get_option
from posix_parity import mergerfs_mount
from posix_parity import mergerfs_set_option
def main():
if len(sys.argv) != 2:
print("usage: TEST_cfg_statfs_ignore <mountpoint>", file=sys.stderr)
return 1
mount = sys.argv[1]
try:
orig_statfs = mergerfs_get_option(mount, "statfs")
orig_ignore = mergerfs_get_option(mount, "statfs-ignore")
except (PermissionError, FileNotFoundError, OSError):
return 0
with mergerfs_mount() as (mount, _):
try:
orig_statfs = mergerfs_get_option(mount, "statfs")
orig_ignore = mergerfs_get_option(mount, "statfs-ignore")
except (PermissionError, FileNotFoundError, OSError):
return 0
try:
mergerfs_set_option(mount, "statfs", "base")
try:
mergerfs_set_option(mount, "statfs", "base")
mergerfs_set_option(mount, "statfs-ignore", "none")
s_none = os.statvfs(mount)
mergerfs_set_option(mount, "statfs-ignore", "none")
s_none = os.statvfs(mount)
mergerfs_set_option(mount, "statfs-ignore", "nc")
s_nc = os.statvfs(mount)
mergerfs_set_option(mount, "statfs-ignore", "nc")
s_nc = os.statvfs(mount)
mergerfs_set_option(mount, "statfs-ignore", "ro")
s_ro = os.statvfs(mount)
mergerfs_set_option(mount, "statfs-ignore", "ro")
s_ro = os.statvfs(mount)
# Ignoring additional branch classes should never increase available blocks.
if s_nc.f_bavail > s_none.f_bavail:
return fail(
f"statfs-ignore=nc increased bavail unexpectedly nc={s_nc.f_bavail} none={s_none.f_bavail}"
)
if s_ro.f_bavail > s_nc.f_bavail:
return fail(
f"statfs-ignore=ro increased bavail unexpectedly ro={s_ro.f_bavail} nc={s_nc.f_bavail}"
)
if s_nc.f_bavail > s_none.f_bavail:
return fail(
f"statfs-ignore=nc increased bavail unexpectedly nc={s_nc.f_bavail} none={s_none.f_bavail}"
)
if s_ro.f_bavail > s_nc.f_bavail:
return fail(
f"statfs-ignore=ro increased bavail unexpectedly ro={s_ro.f_bavail} nc={s_nc.f_bavail}"
)
mergerfs_set_option(mount, "statfs", "full")
s_full = os.statvfs(mount)
if s_full.f_bsize <= 0 or s_full.f_frsize <= 0:
return fail(
f"statfs=full invalid block sizes bsize={s_full.f_bsize} frsize={s_full.f_frsize}"
)
except (PermissionError, FileNotFoundError, OSError):
return 0
finally:
try:
mergerfs_set_option(mount, "statfs", orig_statfs)
mergerfs_set_option(mount, "statfs-ignore", orig_ignore)
except OSError:
pass
mergerfs_set_option(mount, "statfs", "full")
s_full = os.statvfs(mount)
if s_full.f_bsize <= 0 or s_full.f_frsize <= 0:
return fail(
f"statfs=full invalid block sizes bsize={s_full.f_bsize} frsize={s_full.f_frsize}"
)
except (PermissionError, FileNotFoundError, OSError):
return 0
finally:
try:
mergerfs_set_option(mount, "statfs", orig_statfs)
mergerfs_set_option(mount, "statfs-ignore", orig_ignore)
except OSError:
pass
return 0
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+45 -48
View File
@@ -9,6 +9,7 @@ from posix_parity import cleanup_dir
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_get_option
from posix_parity import mergerfs_mount
from posix_parity import mergerfs_set_option
from posix_parity import temp_dir
from posix_parity import touch
@@ -23,58 +24,54 @@ def get_errno(func):
def main():
if len(sys.argv) != 2:
print("usage: TEST_cfg_xattr_modes <mountpoint>", file=sys.stderr)
return 1
mount = sys.argv[1]
merge_base = temp_dir(mount)
try:
path = join(merge_base, "file")
xname = "user.mergerfs.cfg.test"
xval = b"v"
with mergerfs_mount() as (mount, _):
merge_base = temp_dir(mount)
touch(path, b"x")
try:
orig = mergerfs_get_option(mount, "xattr")
except (PermissionError, FileNotFoundError):
return 0
try:
mergerfs_set_option(mount, "xattr", "passthrough")
set_err = get_errno(lambda: os.setxattr(path, xname, xval))
get_err = get_errno(lambda: os.getxattr(path, xname))
if set_err != 0:
return fail(f"xattr=passthrough setxattr expected success got errno={set_err}")
if get_err == errno.ENODATA:
return fail(f"xattr=passthrough getxattr returned ENODATA (known bug): this masks a parity violation")
if get_err != 0:
return fail(f"xattr=passthrough getxattr unexpected errno={get_err}")
mergerfs_set_option(mount, "xattr", "noattr")
err = get_errno(lambda: os.getxattr(path, xname))
if err != errno.ENODATA:
return fail(f"xattr=noattr getxattr expected ENODATA got errno={err}")
err = get_errno(lambda: os.setxattr(path, xname, xval))
if err != errno.ENODATA:
return fail(f"xattr=noattr setxattr expected ENODATA got errno={err}")
# xattr=nosys is intentionally not tested here. It disables the
# xattr API, including the runtime interface this test uses to
# restore options safely.
except (PermissionError, FileNotFoundError):
return 0
finally:
try:
mergerfs_set_option(mount, "xattr", orig)
except OSError:
pass
path = join(merge_base, "file")
xname = "user.test"
xval = b"v"
return 0
finally:
cleanup_dir(merge_base)
touch(path, b"x")
try:
orig = mergerfs_get_option(mount, "xattr")
except (PermissionError, FileNotFoundError):
return 0
try:
mergerfs_set_option(mount, "xattr", "passthrough")
set_err = get_errno(lambda: os.setxattr(path, xname, xval))
get_err = get_errno(lambda: os.getxattr(path, xname))
if set_err != 0:
return fail(f"xattr=passthrough setxattr expected success got errno={set_err}")
if get_err == errno.ENODATA:
return fail(f"xattr=passthrough getxattr returned ENODATA, expected success")
if get_err != 0:
return fail(f"xattr=passthrough getxattr unexpected errno={get_err}")
mergerfs_set_option(mount, "xattr", "noattr")
err = get_errno(lambda: os.getxattr(path, xname))
if err != errno.ENODATA:
return fail(f"xattr=noattr getxattr expected ENODATA got errno={err}")
err = get_errno(lambda: os.setxattr(path, xname, xval))
if err != errno.ENODATA:
return fail(f"xattr=noattr setxattr expected ENODATA got errno={err}")
except (PermissionError, FileNotFoundError):
return 0
finally:
try:
mergerfs_set_option(mount, "xattr", orig)
except OSError:
pass
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+79 -77
View File
@@ -1,102 +1,104 @@
#!/usr/bin/env python3
import tempfile
import os
import sys
import tempfile
def main():
if len(sys.argv) != 2:
print("usage: TEST_io_passthrough_create_open <mountpoint>", file=sys.stderr)
return 1
(fd, filepath) = tempfile.mkstemp(dir=sys.argv[1])
try:
test_data = b"passthrough io test data for create\n" * 100
from posix_parity import mergerfs_mount
with mergerfs_mount() as (mount, _):
(fd, filepath) = tempfile.mkstemp(dir=mount)
bytes_written = os.write(fd, test_data)
if bytes_written != len(test_data):
print("create write failed: expected {} bytes, wrote {}".format(len(test_data), bytes_written))
return 1
try:
test_data = b"passthrough io test data for create\n" * 100
os.lseek(fd, 0, os.SEEK_SET)
read_data = os.read(fd, len(test_data))
if read_data != test_data:
print("create read failed: data mismatch")
return 1
bytes_written = os.write(fd, test_data)
if bytes_written != len(test_data):
print("create write failed: expected {} bytes, wrote {}".format(len(test_data), bytes_written))
return 1
os.close(fd)
fd = -1
os.lseek(fd, 0, os.SEEK_SET)
read_data = os.read(fd, len(test_data))
if read_data != test_data:
print("create read failed: data mismatch")
return 1
fd = os.open(filepath, os.O_RDWR)
os.close(fd)
fd = -1
os.lseek(fd, 0, os.SEEK_SET)
read_data = os.read(fd, len(test_data))
if read_data != test_data:
print("open read failed: data mismatch")
return 1
fd = os.open(filepath, os.O_RDWR)
os.lseek(fd, 0, os.SEEK_END)
more_data = b"additional passthrough data for open\n" * 50
bytes_written = os.write(fd, more_data)
if bytes_written != len(more_data):
print("open write failed: expected {} bytes, wrote {}".format(len(more_data), bytes_written))
return 1
os.lseek(fd, 0, os.SEEK_SET)
read_data = os.read(fd, len(test_data))
if read_data != test_data:
print("open read failed: data mismatch")
return 1
os.lseek(fd, 0, os.SEEK_SET)
all_data = os.read(fd, len(test_data) + len(more_data))
if all_data != test_data + more_data:
print("open final read failed: data mismatch")
return 1
os.lseek(fd, 0, os.SEEK_END)
more_data = b"additional passthrough data for open\n" * 50
bytes_written = os.write(fd, more_data)
if bytes_written != len(more_data):
print("open write failed: expected {} bytes, wrote {}".format(len(more_data), bytes_written))
return 1
fd2 = os.open(filepath, os.O_RDWR)
os.lseek(fd, 0, os.SEEK_SET)
all_data = os.read(fd, len(test_data) + len(more_data))
if all_data != test_data + more_data:
print("open final read failed: data mismatch")
return 1
try:
os.lseek(fd2, 0, os.SEEK_SET)
read_data2 = os.read(fd2, len(test_data) + len(more_data))
if read_data2 != test_data + more_data:
print("second open read failed: data mismatch")
return 1
fd2 = os.open(filepath, os.O_RDWR)
os.lseek(fd2, 0, os.SEEK_END)
extra_data = b"data from second file descriptor\n" * 25
bytes_written = os.write(fd2, extra_data)
if bytes_written != len(extra_data):
print("second open write failed: expected {} bytes, wrote {}".format(len(extra_data), bytes_written))
return 1
try:
os.lseek(fd2, 0, os.SEEK_SET)
read_data2 = os.read(fd2, len(test_data) + len(more_data))
if read_data2 != test_data + more_data:
print("second open read failed: data mismatch")
return 1
os.lseek(fd, 0, os.SEEK_SET)
combined_data = os.read(fd, len(test_data) + len(more_data) + len(extra_data))
expected_data = test_data + more_data + extra_data
if combined_data != expected_data:
print("cross-fd read failed: data mismatch (writes from fd2 not visible on fd)")
return 1
finally:
os.close(fd2)
os.lseek(fd2, 0, os.SEEK_END)
extra_data = b"data from second file descriptor\n" * 25
bytes_written = os.write(fd2, extra_data)
if bytes_written != len(extra_data):
print("second open write failed: expected {} bytes, wrote {}".format(len(extra_data), bytes_written))
return 1
fd3 = os.open(filepath, os.O_RDONLY)
try:
os.lseek(fd3, 0, os.SEEK_SET)
read_data3 = os.read(fd3, len(expected_data))
if read_data3 != expected_data:
print("third open read failed: data mismatch")
return 1
finally:
os.close(fd3)
os.lseek(fd, 0, os.SEEK_SET)
combined_data = os.read(fd, len(test_data) + len(more_data) + len(extra_data))
expected_data = test_data + more_data + extra_data
if combined_data != expected_data:
print("cross-fd read failed: data mismatch (writes from fd2 not visible on fd)")
return 1
finally:
os.close(fd2)
os.close(fd)
fd = -1
fd3 = os.open(filepath, os.O_RDONLY)
try:
os.lseek(fd3, 0, os.SEEK_SET)
read_data3 = os.read(fd3, len(expected_data))
if read_data3 != expected_data:
print("third open read failed: data mismatch")
return 1
finally:
os.close(fd3)
return 0
finally:
if fd >= 0:
os.close(fd)
try:
os.unlink(filepath)
except FileNotFoundError:
pass
os.close(fd)
fd = -1
return 0
finally:
if fd >= 0:
os.close(fd)
try:
os.unlink(filepath)
except FileNotFoundError:
pass
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+77 -74
View File
@@ -5,6 +5,8 @@ import sys
import tempfile
import threading
from posix_parity import mergerfs_mount
NUM_THREADS = 50
TEST_DATA = b"race condition test data\n"
@@ -46,87 +48,88 @@ def open_and_operate(filepath, barrier, results, index):
def main():
if len(sys.argv) != 2:
print("usage: TEST_io_passthrough_open_race <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
(fd, filepath) = tempfile.mkstemp(dir=mount)
(fd, filepath) = tempfile.mkstemp(dir=sys.argv[1])
total_size = NUM_THREADS * len(TEST_DATA)
os.ftruncate(fd, total_size)
os.close(fd)
barrier = threading.Barrier(NUM_THREADS)
results = [None] * NUM_THREADS
threads = []
for i in range(NUM_THREADS):
t = threading.Thread(target=open_and_operate, args=(filepath, barrier, results, i))
threads.append(t)
for t in threads:
t.start()
for t in threads:
t.join()
failed = False
fds_to_close = []
for i, result in enumerate(results):
if result is None:
print("thread {} returned no result".format(i))
failed = True
elif result[0] == "success":
fds_to_close.append(result[1])
else:
print("thread {} failed: {} - {}".format(i, result[0], result[1]))
failed = True
if failed:
for fd in fds_to_close:
total_size = NUM_THREADS * len(TEST_DATA)
os.ftruncate(fd, total_size)
os.close(fd)
os.unlink(filepath)
return 1
if fds_to_close:
verify_fd = fds_to_close[0]
os.lseek(verify_fd, 0, os.SEEK_SET)
all_data = os.read(verify_fd, total_size)
barrier = threading.Barrier(NUM_THREADS)
results = [None] * NUM_THREADS
threads = []
for i in range(NUM_THREADS):
t = threading.Thread(target=open_and_operate, args=(filepath, barrier, results, i))
threads.append(t)
for t in threads:
t.start()
for t in threads:
t.join()
failed = False
fds_to_close = []
for i, result in enumerate(results):
if result is None:
print("thread {} returned no result".format(i))
failed = True
elif result[0] == "success":
fds_to_close.append(result[1])
else:
print("thread {} failed: {} - {}".format(i, result[0], result[1]))
failed = True
if failed:
for fd in fds_to_close:
os.close(fd)
os.unlink(filepath)
return 1
if fds_to_close:
verify_fd = fds_to_close[0]
os.lseek(verify_fd, 0, os.SEEK_SET)
all_data = os.read(verify_fd, total_size)
expected_data = TEST_DATA * NUM_THREADS
if all_data != expected_data:
print("final verification failed: data mismatch")
print("expected {} bytes, got {} bytes".format(len(expected_data), len(all_data)))
for fd in fds_to_close:
os.close(fd)
os.unlink(filepath)
return 1
if len(fds_to_close) >= 2:
fd_a = fds_to_close[0]
fd_b = fds_to_close[1]
new_data = b"cross-fd visibility test\n"
os.lseek(fd_a, 0, os.SEEK_SET)
os.write(fd_a, new_data)
os.lseek(fd_b, 0, os.SEEK_SET)
read_back = os.read(fd_b, len(new_data))
if read_back != new_data:
print("cross-fd visibility failed: write from fd_a not visible on fd_b")
for fd in fds_to_close:
os.close(fd)
os.unlink(filepath)
return 1
expected_data = TEST_DATA * NUM_THREADS
if all_data != expected_data:
print("final verification failed: data mismatch")
print("expected {} bytes, got {} bytes".format(len(expected_data), len(all_data)))
for fd in fds_to_close:
os.close(fd)
os.unlink(filepath)
return 1
if len(fds_to_close) >= 2:
fd_a = fds_to_close[0]
fd_b = fds_to_close[1]
new_data = b"cross-fd visibility test\n"
os.lseek(fd_a, 0, os.SEEK_SET)
os.write(fd_a, new_data)
os.lseek(fd_b, 0, os.SEEK_SET)
read_back = os.read(fd_b, len(new_data))
if read_back != new_data:
print("cross-fd visibility failed: write from fd_a not visible on fd_b")
for fd in fds_to_close:
os.close(fd)
os.unlink(filepath)
return 1
for fd in fds_to_close:
os.close(fd)
os.unlink(filepath)
return 0
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+14 -50
View File
@@ -1,70 +1,34 @@
#!/usr/bin/env python3
import os
import shutil
import subprocess
import sys
import tempfile
import time
from posix_parity import fail
def has_tool(name):
return shutil.which(name) is not None
from posix_parity import find_fusermount
from posix_parity import find_mergerfs
from posix_parity import mergerfs_mount
def main():
if len(sys.argv) != 2:
print("usage: TEST_mount_lifecycle <mountpoint>", file=sys.stderr)
return 1
# This is a soft integration harness check for init/destroy wiring.
# Skip when environment is not prepared for nested mount tests.
if os.geteuid() != 0:
return 0
if not has_tool("mergerfs"):
if find_mergerfs() is None:
return 0
if not has_tool("fusermount") and not has_tool("fusermount3"):
if find_fusermount() is None:
return 0
with tempfile.TemporaryDirectory() as td:
b1 = os.path.join(td, "b1")
b2 = os.path.join(td, "b2")
mp = os.path.join(td, "mp")
os.makedirs(b1, exist_ok=True)
os.makedirs(b2, exist_ok=True)
os.makedirs(mp, exist_ok=True)
cmd = [
"mergerfs",
f"{b1}:{b2}",
mp,
"-o",
"defaults,allow_other,use_ino,category.create=mfs",
]
p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if p.returncode != 0:
return fail(f"mount lifecycle: mergerfs mount failed: {p.stderr.decode(errors='replace')}")
try:
with open(os.path.join(mp, "lifecycle-file"), "wb") as fp:
try:
with mergerfs_mount() as (mount, _):
with open(os.path.join(mount, "lifecycle-file"), "wb") as fp:
fp.write(b"ok")
if not os.path.exists(os.path.join(mp, "lifecycle-file")):
if not os.path.exists(os.path.join(mount, "lifecycle-file")):
return fail("mount lifecycle: expected created file to exist")
finally:
um = shutil.which("fusermount3") or shutil.which("fusermount")
for attempt in range(3):
u = subprocess.run([um, "-u", mp], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if u.returncode == 0:
break
time.sleep(1)
if u.returncode != 0:
subprocess.run([um, "-z", mp], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
return 0
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+49 -52
View File
@@ -9,6 +9,7 @@ from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_branches
from posix_parity import mergerfs_get_option
from posix_parity import mergerfs_mount
from posix_parity import mergerfs_set_option
from posix_parity import temp_dir
from posix_parity import touch
@@ -18,70 +19,66 @@ SKIP = 77
def main():
if len(sys.argv) != 2:
print("usage: TEST_movefile_enospc <mountpoint>", file=sys.stderr)
return 1
mount = sys.argv[1]
try:
orig_moveonenospc = mergerfs_get_option(mount, "moveonenospc")
except (PermissionError, FileNotFoundError):
print("moveonenospc runtime option unavailable", end="")
return SKIP
except OSError as exc:
if exc.errno == errno.EOPNOTSUPP:
print("moveonenospc runtime option unsupported", end="")
return SKIP
raise
with mergerfs_mount(num_branches=2) as (mount, branches):
try:
orig_moveonenospc = mergerfs_get_option(mount, "moveonenospc")
except (PermissionError, FileNotFoundError):
print("moveonenospc runtime option unavailable", end="")
return SKIP
except OSError as exc:
if exc.errno == errno.EOPNOTSUPP:
print("moveonenospc runtime option unsupported", end="")
return SKIP
raise
branches = mergerfs_branches(mount)
if len(branches) < 2:
print("moveonenospc test requires at least 2 branches", end="")
return SKIP
if len(branches) < 2:
print("moveonenospc test requires at least 2 branches", end="")
return SKIP
# The documented feature only runs after write returns ENOSPC/EDQUOT. This
# generic mountpoint test cannot safely force a branch into that state.
merge_base = temp_dir(mount)
merge_base = temp_dir(mount)
try:
mergerfs_set_option(mount, "moveonenospc", "mfs")
if mergerfs_get_option(mount, "moveonenospc") != "mfs":
return fail("moveonenospc: failed to set documented policy value")
try:
mergerfs_set_option(mount, "moveonenospc", "mfs")
if mergerfs_get_option(mount, "moveonenospc") != "mfs":
return fail("moveonenospc: failed to set documented policy value")
merge_file = join(merge_base, "enospc_test")
touch(merge_file, b"test data for enospc")
merge_file = join(merge_base, "enospc_test")
touch(merge_file, b"test data for enospc")
if not os.path.exists(merge_file):
return fail("moveonenospc: file was not created")
if not os.path.exists(merge_file):
return fail("moveonenospc: file was not created")
fullpath = None
try:
fullpath = os.getxattr(merge_file, "user.mergerfs.fullpath").decode()
except OSError:
pass
fullpath = None
try:
fullpath = os.getxattr(merge_file, "user.mergerfs.fullpath").decode()
except OSError:
pass
if fullpath and not os.path.exists(fullpath):
return fail(f"moveonenospc: fullpath does not exist on disk: {fullpath}")
if fullpath and not os.path.exists(fullpath):
return fail(f"moveonenospc: fullpath does not exist on disk: {fullpath}")
with open(merge_file, "r") as f:
content = f.read()
with open(merge_file, "r") as f:
content = f.read()
if content != "test data for enospc":
return fail(f"moveonenospc: file content mismatch: got {content!r}")
if content != "test data for enospc":
return fail(f"moveonenospc: file content mismatch: got {content!r}")
mergerfs_set_option(mount, "moveonenospc", orig_moveonenospc)
mergerfs_set_option(mount, "moveonenospc", orig_moveonenospc)
print("moveonenospc ENOSPC path requires a constrained branch", end="")
return SKIP
except Exception as exc:
try:
mergerfs_set_option(mount, "moveonenospc", orig_moveonenospc)
except Exception:
pass
return fail(str(exc))
finally:
cleanup_dir(merge_base)
print("moveonenospc ENOSPC path requires a constrained branch", end="")
return SKIP
except Exception as exc:
try:
mergerfs_set_option(mount, "moveonenospc", orig_moveonenospc)
except Exception:
pass
return fail(str(exc))
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+20 -13
View File
@@ -1,27 +1,34 @@
#!/usr/bin/env python3
import tempfile
import os
import sys
import tempfile
from posix_parity import mergerfs_mount
def main():
(fd, filepath) = tempfile.mkstemp(dir=sys.argv[1])
try:
with mergerfs_mount() as (mount, _):
(fd, filepath) = tempfile.mkstemp(dir=mount)
os.unlink(filepath)
os.unlink(filepath)
found = False
for entry in os.scandir(sys.argv[1]):
if entry.name.startswith(".fuse_hidden"):
print("found .fuse_hidden file {}".format(entry.name), end='')
found = True
found = False
for entry in os.scandir(mount):
if entry.name.startswith(".fuse_hidden"):
print("found .fuse_hidden file {}".format(entry.name), end='')
found = True
os.close(fd)
os.close(fd)
if found:
return 1
return 0
if found:
return 1
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+38 -31
View File
@@ -1,11 +1,13 @@
#!/usr/bin/env python3
import tempfile
import ctypes
import mmap
import os
import resource
import sys
import tempfile
from posix_parity import mergerfs_mount
libc = ctypes.CDLL(None, use_errno=True)
@@ -28,38 +30,43 @@ def aligned_read(fd, size):
def main():
(fd, filepath) = tempfile.mkstemp(dir=sys.argv[1])
os.close(fd)
fd = os.open(filepath, os.O_RDWR|os.O_DIRECT|os.O_TRUNC)
os.unlink(filepath)
size = resource.getpagesize()
pattern = (b"mergerfs-o-direct" * ((size // 16) + 1))[:size]
buf = mmap.mmap(-1, size)
buf.write(pattern)
buf.seek(0)
try:
written = os.write(fd, buf)
if written != size:
print(f"O_DIRECT short write: wrote {written}, expected {size}", end="")
return 1
os.lseek(fd, 0, os.SEEK_SET)
data = aligned_read(fd, size)
if len(data) != size:
print(f"O_DIRECT short read: read {len(data)}, expected {size}", end="")
return 1
if data != pattern:
print("O_DIRECT data mismatch", end="")
return 1
finally:
buf.close()
os.close(fd)
with mergerfs_mount() as (mount, _):
(fd, filepath) = tempfile.mkstemp(dir=mount)
return 0
os.close(fd)
fd = os.open(filepath, os.O_RDWR|os.O_DIRECT|os.O_TRUNC)
os.unlink(filepath)
size = resource.getpagesize()
pattern = (b"mergerfs-o-direct" * ((size // 16) + 1))[:size]
buf = mmap.mmap(-1, size)
buf.write(pattern)
buf.seek(0)
try:
written = os.write(fd, buf)
if written != size:
print(f"O_DIRECT short write: wrote {written}, expected {size}", end="")
return 1
os.lseek(fd, 0, os.SEEK_SET)
data = aligned_read(fd, size)
if len(data) != size:
print(f"O_DIRECT short read: read {len(data)}, expected {size}", end="")
return 1
if data != pattern:
print("O_DIRECT data mismatch", end="")
return 1
finally:
buf.close()
os.close(fd)
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+46 -46
View File
@@ -2,67 +2,67 @@
import os
import sys
import tempfile
from posix_parity import cleanup_dir
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def main():
if len(sys.argv) != 2:
print("usage: TEST_open_after_unlink <mountpoint>", file=sys.stderr)
return 1
mount = sys.argv[1]
merge_base = temp_dir(mount)
try:
merge_file = join(merge_base, "testfile")
touch(merge_file, b"hello world")
with mergerfs_mount() as (mount, _):
merge_base = temp_dir(mount)
fd = os.open(merge_file, os.O_RDWR)
try:
data = os.read(fd, 4096)
if data != b"hello world":
return fail("read before unlink: expected 'hello world', got {!r}".format(data))
os.unlink(merge_file)
fd2 = os.dup(fd)
try:
os.lseek(fd2, 0, os.SEEK_SET)
data2 = os.read(fd2, 4096)
if data2 != b"hello world":
return fail("read after unlink via dup: expected 'hello world', got {!r}".format(data2))
merge_file = join(merge_base, "testfile")
touch(merge_file, b"hello world")
os.lseek(fd, 0, os.SEEK_SET)
written = os.write(fd, b"modified")
if written != len(b"modified"):
return fail("write after unlink: wrote {}, expected {}".format(written, len(b"modified")))
fd = os.open(merge_file, os.O_RDWR)
os.lseek(fd, 0, os.SEEK_SET)
data3 = os.read(fd, 4096)
if not data3.startswith(b"modified"):
return fail("read back written data: got {!r} after writing 'modified'".format(data3))
try:
data = os.read(fd, 4096)
if data != b"hello world":
return fail("read before unlink: expected 'hello world', got {!r}".format(data))
os.unlink(merge_file)
fd2 = os.dup(fd)
try:
os.lseek(fd2, 0, os.SEEK_SET)
data2 = os.read(fd2, 4096)
if data2 != b"hello world":
return fail("read after unlink via dup: expected 'hello world', got {!r}".format(data2))
os.lseek(fd, 0, os.SEEK_SET)
written = os.write(fd, b"modified")
if written != len(b"modified"):
return fail("write after unlink: wrote {}, expected {}".format(written, len(b"modified")))
os.lseek(fd, 0, os.SEEK_SET)
data3 = os.read(fd, 4096)
if not data3.startswith(b"modified"):
return fail("read back written data: got {!r} after writing 'modified'".format(data3))
finally:
os.close(fd2)
finally:
os.close(fd)
return 0
except Exception as exc:
return fail(str(exc))
finally:
os.close(fd2)
finally:
os.close(fd)
return 0
except Exception as exc:
return fail(str(exc))
finally:
try:
os.unlink(join(merge_base, "testfile"))
except FileNotFoundError:
pass
cleanup_dir(merge_base)
try:
os.unlink(join(merge_base, "testfile"))
except FileNotFoundError:
pass
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+57 -58
View File
@@ -8,6 +8,7 @@ from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_branches
from posix_parity import mergerfs_get_option
from posix_parity import mergerfs_mount
from posix_parity import mergerfs_set_option
from posix_parity import temp_dir
from posix_parity import touch
@@ -50,75 +51,73 @@ def least_used_branches(branches, rel_dir):
def main():
if len(sys.argv) != 2:
print("usage: TEST_policy_lup <mountpoint>", file=sys.stderr)
return 1
mount = sys.argv[1]
try:
orig_create = mergerfs_get_option(mount, "func.create")
except (PermissionError, FileNotFoundError, OSError):
print("LUP create policy option unavailable", end="")
return SKIP
with mergerfs_mount(num_branches=2) as (mount, branches):
try:
orig_create = mergerfs_get_option(mount, "func.create")
except (PermissionError, FileNotFoundError, OSError):
print("LUP create policy option unavailable", end="")
return SKIP
try:
orig_search = mergerfs_get_option(mount, "func.search")
except (PermissionError, FileNotFoundError, OSError):
print("LUP search policy option unavailable", end="")
return SKIP
try:
orig_search = mergerfs_get_option(mount, "func.search")
except (PermissionError, FileNotFoundError, OSError):
print("LUP search policy option unavailable", end="")
return SKIP
branches = mergerfs_branches(mount)
if len(branches) < 2:
print("LUP policy test requires at least 2 branches", end="")
return SKIP
if len(branches) < 2:
print("LUP policy test requires at least 2 branches", end="")
return SKIP
merge_base = temp_dir(mount)
rel_base = os.path.relpath(merge_base, mount)
merge_base = temp_dir(mount)
rel_base = os.path.relpath(merge_base, mount)
try:
for branch in branches:
os.makedirs(join(branch, rel_base), exist_ok=True)
try:
for branch in branches:
os.makedirs(join(branch, rel_base), exist_ok=True)
mergerfs_set_option(mount, "func.create", "lup")
mergerfs_set_option(mount, "func.search", "lup")
mergerfs_set_option(mount, "func.create", "lup")
mergerfs_set_option(mount, "func.search", "lup")
path = join(merge_base, "lup_test_file")
expected = least_used_branches(branches, rel_base)
touch(path, b"lup create test")
if not os.path.exists(path):
return fail("LUP create: file not created")
path = join(merge_base, "lup_test_file")
expected = least_used_branches(branches, rel_base)
touch(path, b"lup create test")
if not os.path.exists(path):
return fail("LUP create: file not created")
fullpath = os.getxattr(path, "user.mergerfs.fullpath").decode()
if not fullpath:
return fail("LUP create: no fullpath attribute")
actual = branch_for_fullpath(branches, fullpath)
if actual not in expected:
return fail(f"LUP create: selected {actual}, expected one of {sorted(expected)}")
fullpath = os.getxattr(path, "user.mergerfs.fullpath").decode()
if not fullpath:
return fail("LUP create: no fullpath attribute")
actual = branch_for_fullpath(branches, fullpath)
if actual not in expected:
return fail(f"LUP create: selected {actual}, expected one of {sorted(expected)}")
path2 = join(merge_base, "lup_test_file2")
for branch in branches:
touch(join(branch, rel_base, "lup_test_file2"), b"lup search test")
expected = least_used_branches(branches, rel_base)
fullpath2 = os.getxattr(path2, "user.mergerfs.fullpath").decode()
if not fullpath2:
return fail("LUP search: no fullpath attribute")
actual = branch_for_fullpath(branches, fullpath2)
if actual not in expected:
return fail(f"LUP search: selected {actual}, expected one of {sorted(expected)}")
path2 = join(merge_base, "lup_test_file2")
for branch in branches:
touch(join(branch, rel_base, "lup_test_file2"), b"lup search test")
expected = least_used_branches(branches, rel_base)
fullpath2 = os.getxattr(path2, "user.mergerfs.fullpath").decode()
if not fullpath2:
return fail("LUP search: no fullpath attribute")
actual = branch_for_fullpath(branches, fullpath2)
if actual not in expected:
return fail(f"LUP search: selected {actual}, expected one of {sorted(expected)}")
mergerfs_set_option(mount, "func.create", orig_create)
mergerfs_set_option(mount, "func.search", orig_search)
mergerfs_set_option(mount, "func.create", orig_create)
mergerfs_set_option(mount, "func.search", orig_search)
return 0
except Exception as exc:
mergerfs_set_option(mount, "func.create", orig_create)
mergerfs_set_option(mount, "func.search", orig_search)
return fail(str(exc))
finally:
cleanup_dir(merge_base)
for branch in branches:
cleanup_dir(join(branch, rel_base))
return 0
except Exception as exc:
mergerfs_set_option(mount, "func.create", orig_create)
mergerfs_set_option(mount, "func.search", orig_search)
return fail(str(exc))
finally:
cleanup_dir(merge_base)
for branch in branches:
cleanup_dir(join(branch, rel_base))
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+49 -49
View File
@@ -10,70 +10,70 @@ from posix_parity import cleanup_paths
from posix_parity import compare_access
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_access <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
cleanup_paths([merge_file, merge_notdir])
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
touch(merge_file, b"ok", 0o644)
touch(native_file, b"ok", 0o644)
touch(merge_notdir, b"x", 0o644)
touch(native_notdir, b"x", 0o644)
cleanup_paths([merge_file, merge_notdir])
err = compare_access("F_OK existing", merge_file, native_file, os.F_OK)
if err:
return fail(err)
touch(merge_file, b"ok", 0o644)
touch(native_file, b"ok", 0o644)
touch(merge_notdir, b"x", 0o644)
touch(native_notdir, b"x", 0o644)
err = compare_access(
"F_OK missing", merge_missing, native_missing, os.F_OK, errno.ENOENT
)
if err:
return fail(err)
err = compare_access("F_OK existing", merge_file, native_file, os.F_OK)
if err:
return fail(err)
err = compare_access(
"X_OK non-directory prefix",
join(merge_notdir, "child"),
join(native_notdir, "child"),
os.X_OK,
errno.ENOTDIR,
)
if err:
return fail(err)
err = compare_access(
"F_OK missing", merge_missing, native_missing, os.F_OK, errno.ENOENT
)
if err:
return fail(err)
os.chmod(merge_file, 0)
os.chmod(native_file, 0)
err = compare_access(
"X_OK non-directory prefix",
join(merge_notdir, "child"),
join(native_notdir, "child"),
os.X_OK,
errno.ENOTDIR,
)
if err:
return fail(err)
err = compare_access("R_OK unreadable", merge_file, native_file, os.R_OK)
if err:
return fail(err)
os.chmod(merge_file, 0)
os.chmod(native_file, 0)
os.chmod(merge_file, 0o644)
os.chmod(native_file, 0o644)
err = compare_access("R_OK unreadable", merge_file, native_file, os.R_OK)
if err:
return fail(err)
os.chmod(merge_file, 0o644)
os.chmod(native_file, 0o644)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+40 -42
View File
@@ -1,5 +1,6 @@
#!/usr/bin/env python3
import ctypes
import errno
import fcntl
import os
@@ -11,6 +12,7 @@ from posix_parity import cleanup_dir
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
@@ -25,52 +27,48 @@ def fibmap(fd, block=0):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_bmap <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
touch(merge_file, b"x" * 8192)
touch(native_file, b"x" * 8192)
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
try:
err = compare_calls(
"ioctl FIBMAP block0",
lambda: fibmap(mfd, 0),
lambda: fibmap(nfd, 0),
lambda a, b: (a >= 0) == (b >= 0),
)
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
touch(merge_file, b"x" * 8192)
touch(native_file, b"x" * 8192)
err = compare_calls(
"ioctl FIBMAP EBADF",
lambda: fibmap(mfd, 0),
lambda: fibmap(nfd, 0),
)
if err:
return fail(err)
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
try:
# FIBMAP often requires CAP_SYS_RAWIO; compare errno parity if denied.
err = compare_calls(
"ioctl FIBMAP block0",
lambda: fibmap(mfd, 0),
lambda: fibmap(nfd, 0),
lambda a, b: (a >= 0) == (b >= 0),
)
if err:
# If both denied by privilege checks, compare_calls already passes.
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
# EBADF parity on closed fds.
err = compare_calls(
"ioctl FIBMAP EBADF",
lambda: fibmap(mfd, 0),
lambda: fibmap(nfd, 0),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+65 -64
View File
@@ -11,6 +11,7 @@ from posix_parity import cleanup_paths
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import should_compare_inode
from posix_parity import stat_cmp_basic
from posix_parity import temp_dir
@@ -22,80 +23,80 @@ def stat_cmp_basic_with_inode(lhs, rhs):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_chmod <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
stcmp = stat_cmp_basic_with_inode if should_compare_inode(mount) else stat_cmp_basic
mount = sys.argv[1]
stcmp = stat_cmp_basic_with_inode if should_compare_inode(mount) else stat_cmp_basic
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
cleanup_paths([merge_file, merge_notdir])
cleanup_paths([merge_file, merge_notdir])
touch(merge_file, b"x", 0o644)
touch(native_file, b"x", 0o644)
touch(merge_notdir, b"x", 0o644)
touch(native_notdir, b"x", 0o644)
touch(merge_file, b"x", 0o644)
touch(native_file, b"x", 0o644)
touch(merge_notdir, b"x", 0o644)
touch(native_notdir, b"x", 0o644)
err = compare_calls(
"chmod success",
lambda: os.chmod(merge_file, 0o600),
lambda: os.chmod(native_file, 0o600),
)
if err:
return fail(err)
err = compare_calls(
"chmod success",
lambda: os.chmod(merge_file, 0o600),
lambda: os.chmod(native_file, 0o600),
)
if err:
return fail(err)
err = compare_calls(
"chmod stat parity",
lambda: os.lstat(merge_file),
lambda: os.lstat(native_file),
stcmp,
)
if err:
return fail(err)
err = compare_calls(
"chmod stat parity",
lambda: os.lstat(merge_file),
lambda: os.lstat(native_file),
stcmp,
)
if err:
return fail(err)
err = compare_calls(
"chmod ENOENT",
lambda: os.chmod(merge_missing, 0o600),
lambda: os.chmod(native_missing, 0o600),
)
if err:
return fail(err)
err = compare_calls(
"chmod ENOENT",
lambda: os.chmod(merge_missing, 0o600),
lambda: os.chmod(native_missing, 0o600),
)
if err:
return fail(err)
err = compare_calls(
"chmod ENOTDIR",
lambda: os.chmod(join(merge_notdir, "child"), 0o600),
lambda: os.chmod(join(native_notdir, "child"), 0o600),
)
if err:
return fail(err)
err = compare_calls(
"chmod ENOTDIR",
lambda: os.chmod(join(merge_notdir, "child"), 0o600),
lambda: os.chmod(join(native_notdir, "child"), 0o600),
)
if err:
return fail(err)
if os.geteuid() != 0:
err = compare_calls(
"chmod setuid parity",
lambda: os.chmod(merge_file, stat.S_ISUID | 0o644),
lambda: os.chmod(native_file, stat.S_ISUID | 0o644),
)
if err:
return fail(err)
else:
_ = errno.EPERM
if os.geteuid() != 0:
err = compare_calls(
"chmod setuid parity",
lambda: os.chmod(merge_file, stat.S_ISUID | 0o644),
lambda: os.chmod(native_file, stat.S_ISUID | 0o644),
)
if err:
return fail(err)
else:
_ = errno.EPERM
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+56 -55
View File
@@ -9,75 +9,76 @@ from posix_parity import cleanup_paths
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_chown <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
uid = os.getuid()
gid = os.getgid()
mount = sys.argv[1]
uid = os.getuid()
gid = os.getgid()
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
cleanup_paths([merge_file, merge_notdir])
cleanup_paths([merge_file, merge_notdir])
touch(merge_file, b"x", 0o644)
touch(native_file, b"x", 0o644)
touch(merge_notdir, b"x", 0o644)
touch(native_notdir, b"x", 0o644)
touch(merge_file, b"x", 0o644)
touch(native_file, b"x", 0o644)
touch(merge_notdir, b"x", 0o644)
touch(native_notdir, b"x", 0o644)
err = compare_calls(
"chown self",
lambda: os.chown(merge_file, uid, gid),
lambda: os.chown(native_file, uid, gid),
)
if err:
return fail(err)
err = compare_calls(
"chown self",
lambda: os.chown(merge_file, uid, gid),
lambda: os.chown(native_file, uid, gid),
)
if err:
return fail(err)
err = compare_calls(
"chown ENOENT",
lambda: os.chown(merge_missing, uid, gid),
lambda: os.chown(native_missing, uid, gid),
)
if err:
return fail(err)
err = compare_calls(
"chown ENOENT",
lambda: os.chown(merge_missing, uid, gid),
lambda: os.chown(native_missing, uid, gid),
)
if err:
return fail(err)
err = compare_calls(
"chown ENOTDIR",
lambda: os.chown(join(merge_notdir, "child"), uid, gid),
lambda: os.chown(join(native_notdir, "child"), uid, gid),
)
if err:
return fail(err)
err = compare_calls(
"chown ENOTDIR",
lambda: os.chown(join(merge_notdir, "child"), uid, gid),
lambda: os.chown(join(native_notdir, "child"), uid, gid),
)
if err:
return fail(err)
if os.geteuid() != 0:
err = compare_calls(
"chown EPERM foreign uid",
lambda: os.chown(merge_file, 0, gid),
lambda: os.chown(native_file, 0, gid),
)
if err:
return fail(err)
if os.geteuid() != 0:
err = compare_calls(
"chown EPERM foreign uid",
lambda: os.chown(merge_file, 0, gid),
lambda: os.chown(native_file, 0, gid),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+159 -115
View File
@@ -1,7 +1,9 @@
#!/usr/bin/env python3
import errno
import hashlib
import os
import shutil
import sys
import tempfile
@@ -9,131 +11,173 @@ from posix_parity import cleanup_dir
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def _native_tmp():
tests_dir = os.path.dirname(os.path.realpath(__file__))
parent = os.path.join(tests_dir, ".test_tmp")
os.makedirs(parent, exist_ok=True)
return tempfile.mkdtemp(prefix="native_", dir=parent)
class NativeTmp:
def __enter__(self):
self.path = _native_tmp()
return self.path
def __exit__(self, *args):
shutil.rmtree(self.path, ignore_errors=True)
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_copy_file_range <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
if not hasattr(os, "copy_file_range"):
return 0
if not hasattr(os, "copy_file_range"):
return 0
mount = sys.argv[1]
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
merge_src = join(merge_base, "src")
merge_dst = join(merge_base, "dst")
native_src = join(native_base, "src")
native_dst = join(native_base, "dst")
payload = b"0123456789abcdefghijklmnopqrstuvwxyz"
touch(merge_src, payload)
touch(native_src, payload)
touch(merge_dst, b"")
touch(native_dst, b"")
msfd = os.open(merge_src, os.O_RDONLY)
mdfd = os.open(merge_dst, os.O_WRONLY)
nsfd = os.open(native_src, os.O_RDONLY)
ndfd = os.open(native_dst, os.O_WRONLY)
try:
err = compare_calls(
"copy_file_range success",
lambda: os.copy_file_range(msfd, mdfd, 16),
lambda: os.copy_file_range(nsfd, ndfd, 16),
lambda a, b: a == b,
)
if err:
return fail(err)
finally:
os.close(msfd)
os.close(mdfd)
os.close(nsfd)
os.close(ndfd)
with open(merge_dst, "rb") as mf, open(native_dst, "rb") as nf:
mdata = mf.read()
ndata = nf.read()
if mdata != ndata:
return fail(f"copy_file_range dst mismatch mergerfs={mdata!r} native={ndata!r}")
mdfd = os.open(merge_dst, os.O_WRONLY)
ndfd = os.open(native_dst, os.O_WRONLY)
try:
err = compare_calls(
"copy_file_range EBADF src",
lambda: os.copy_file_range(-1, mdfd, 1),
lambda: os.copy_file_range(-1, ndfd, 1),
)
if err:
return fail(err)
finally:
os.close(mdfd)
os.close(ndfd)
# Exercise very large copy lengths to validate the 64-bit copy result
# path when available. Use sparse files to keep disk usage low.
# NOTE: 5GB+ length may be slow on some filesystems; targets
# ENOSPC/EFBIG early if space is insufficient.
large_len = (5 << 30) + 4096
merge_large_src = join(merge_base, "src-large")
merge_large_dst = join(merge_base, "dst-large")
native_large_src = join(native_base, "src-large")
native_large_dst = join(native_base, "dst-large")
touch(merge_large_src, b"")
touch(merge_large_dst, b"")
touch(native_large_src, b"")
touch(native_large_dst, b"")
msfd = os.open(merge_large_src, os.O_RDWR)
mdfd = os.open(merge_large_dst, os.O_RDWR)
nsfd = os.open(native_large_src, os.O_RDWR)
ndfd = os.open(native_large_dst, os.O_RDWR)
try:
with NativeTmp() as native:
merge_base = temp_dir(mount)
try:
os.ftruncate(msfd, large_len)
os.ftruncate(nsfd, large_len)
os.pwrite(msfd, b"z", large_len - 1)
os.pwrite(nsfd, b"z", large_len - 1)
except OSError as exc:
if exc.errno in (errno.ENOSPC, errno.EFBIG):
return 0
raise
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
err = compare_calls(
"copy_file_range large sparse",
lambda: os.copy_file_range(msfd, mdfd, large_len),
lambda: os.copy_file_range(nsfd, ndfd, large_len),
lambda a, b: a == b,
)
if err:
return fail(err)
merge_src = join(merge_base, "src")
merge_dst = join(merge_base, "dst")
native_src = join(native_base, "src")
native_dst = join(native_base, "dst")
mstat = os.fstat(mdfd)
nstat = os.fstat(ndfd)
if mstat.st_size != nstat.st_size:
return fail(
"copy_file_range large sparse dst size mismatch "
f"mergerfs={mstat.st_size} native={nstat.st_size}"
)
finally:
os.close(msfd)
os.close(mdfd)
os.close(nsfd)
os.close(ndfd)
payload = b"0123456789abcdefghijklmnopqrstuvwxyz"
touch(merge_src, payload)
touch(native_src, payload)
touch(merge_dst, b"")
touch(native_dst, b"")
return 0
finally:
cleanup_dir(merge_base)
msfd = os.open(merge_src, os.O_RDONLY)
mdfd = os.open(merge_dst, os.O_WRONLY)
nsfd = os.open(native_src, os.O_RDONLY)
ndfd = os.open(native_dst, os.O_WRONLY)
try:
err = compare_calls(
"copy_file_range success",
lambda: os.copy_file_range(msfd, mdfd, 16),
lambda: os.copy_file_range(nsfd, ndfd, 16),
lambda a, b: a == b,
)
if err:
return fail(err)
finally:
os.close(msfd)
os.close(mdfd)
os.close(nsfd)
os.close(ndfd)
with open(merge_dst, "rb") as mf, open(native_dst, "rb") as nf:
mdata = mf.read()
ndata = nf.read()
if mdata != ndata:
return fail(f"copy_file_range dst mismatch mergerfs={mdata!r} native={ndata!r}")
mdfd = os.open(merge_dst, os.O_WRONLY)
ndfd = os.open(native_dst, os.O_WRONLY)
try:
err = compare_calls(
"copy_file_range EBADF src",
lambda: os.copy_file_range(-1, mdfd, 1),
lambda: os.copy_file_range(-1, ndfd, 1),
)
if err:
return fail(err)
finally:
os.close(mdfd)
os.close(ndfd)
large_len = (5 << 30) + 4096
merge_large_src = join(merge_base, "src-large")
merge_large_dst = join(merge_base, "dst-large")
native_large_src = join(native_base, "src-large")
native_large_dst = join(native_base, "dst-large")
touch(merge_large_src, b"")
touch(merge_large_dst, b"")
touch(native_large_src, b"")
touch(native_large_dst, b"")
msfd = os.open(merge_large_src, os.O_RDWR)
mdfd = os.open(merge_large_dst, os.O_RDWR)
nsfd = os.open(native_large_src, os.O_RDWR)
ndfd = os.open(native_large_dst, os.O_RDWR)
try:
try:
os.ftruncate(msfd, large_len)
os.ftruncate(nsfd, large_len)
os.pwrite(msfd, b"z", large_len - 1)
os.pwrite(nsfd, b"z", large_len - 1)
except OSError as exc:
if exc.errno in (errno.ENOSPC, errno.EFBIG):
return 0
raise
total_m = 0
while True:
n = os.copy_file_range(msfd, mdfd, large_len)
if n == 0:
break
total_m += n
if total_m >= large_len:
break
total_n = 0
while True:
n = os.copy_file_range(nsfd, ndfd, large_len)
if n == 0:
break
total_n += n
if total_n >= large_len:
break
mstat = os.fstat(mdfd)
nstat = os.fstat(ndfd)
if mstat.st_size != nstat.st_size:
return fail(
"copy_file_range large sparse dst size mismatch "
f"mergerfs={mstat.st_size} native={nstat.st_size}"
)
def sha256_partial(fd, size):
h = hashlib.sha256()
pos = os.lseek(fd, 0, os.SEEK_SET)
remaining = size
while remaining > 0:
chunk = os.read(fd, min(remaining, 65536))
if not chunk:
break
h.update(chunk)
remaining -= len(chunk)
os.lseek(fd, pos, os.SEEK_SET)
return h.digest()
if os.path.getsize(merge_large_dst) != os.path.getsize(native_large_dst):
return fail("copy_file_range large: final sizes differ")
mhash = sha256_partial(mdfd, mstat.st_size)
nhash = sha256_partial(ndfd, nstat.st_size)
if mhash != nhash:
return fail("copy_file_range large: content hash mismatch")
finally:
os.close(msfd)
os.close(mdfd)
os.close(nsfd)
os.close(ndfd)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+69 -69
View File
@@ -10,90 +10,90 @@ from posix_parity import cleanup_paths
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_create_mknod <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_exist = join(merge_base, "exist")
native_exist = join(native_base, "exist")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
cleanup_paths([merge_file, merge_exist, merge_notdir])
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_exist = join(merge_base, "exist")
native_exist = join(native_base, "exist")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
err = compare_calls(
"create success",
lambda: os.open(merge_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o640),
lambda: os.open(native_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o640),
close_fds=True,
)
if err:
return fail(err)
cleanup_paths([merge_file, merge_exist, merge_notdir])
with open(merge_exist, "wb"):
pass
with open(native_exist, "wb"):
pass
err = compare_calls(
"create success",
lambda: os.open(merge_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o640),
lambda: os.open(native_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o640),
close_fds=True,
)
if err:
return fail(err)
err = compare_calls(
"create EEXIST",
lambda: os.open(merge_exist, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o640),
lambda: os.open(native_exist, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o640),
close_fds=True,
)
if err:
return fail(err)
with open(merge_exist, "wb"):
pass
with open(native_exist, "wb"):
pass
with open(merge_notdir, "wb"):
pass
with open(native_notdir, "wb"):
pass
err = compare_calls(
"create EEXIST",
lambda: os.open(merge_exist, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o640),
lambda: os.open(native_exist, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o640),
close_fds=True,
)
if err:
return fail(err)
err = compare_calls(
"create ENOTDIR",
lambda: os.open(join(merge_notdir, "child"), os.O_CREAT | os.O_WRONLY, 0o640),
lambda: os.open(join(native_notdir, "child"), os.O_CREAT | os.O_WRONLY, 0o640),
)
if err:
return fail(err)
with open(merge_notdir, "wb"):
pass
with open(native_notdir, "wb"):
pass
merge_node = join(merge_base, "mknod")
native_node = join(native_base, "mknod")
cleanup_paths([merge_node])
err = compare_calls(
"create ENOTDIR",
lambda: os.open(join(merge_notdir, "child"), os.O_CREAT | os.O_WRONLY, 0o640),
lambda: os.open(join(native_notdir, "child"), os.O_CREAT | os.O_WRONLY, 0o640),
)
if err:
return fail(err)
err = compare_calls(
"mknod regular",
lambda: os.mknod(merge_node, stat.S_IFREG | 0o600, 0),
lambda: os.mknod(native_node, stat.S_IFREG | 0o600, 0),
)
if err:
return fail(err)
merge_node = join(merge_base, "mknod")
native_node = join(native_base, "mknod")
cleanup_paths([merge_node])
err = compare_calls(
"mknod EEXIST",
lambda: os.mknod(merge_node, stat.S_IFREG | 0o600, 0),
lambda: os.mknod(native_node, stat.S_IFREG | 0o600, 0),
)
if err:
return fail(err)
err = compare_calls(
"mknod regular",
lambda: os.mknod(merge_node, stat.S_IFREG | 0o600, 0),
lambda: os.mknod(native_node, stat.S_IFREG | 0o600, 0),
)
if err:
return fail(err)
err = compare_calls(
"mknod EEXIST",
lambda: os.mknod(merge_node, stat.S_IFREG | 0o600, 0),
lambda: os.mknod(native_node, stat.S_IFREG | 0o600, 0),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+56 -57
View File
@@ -8,76 +8,75 @@ from posix_parity import cleanup_dir
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_fsync_flush <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
touch(merge_file, b"abc")
touch(native_file, b"abc")
os.makedirs(merge_dir, exist_ok=True)
os.makedirs(native_dir, exist_ok=True)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
mfd = os.open(merge_file, os.O_RDWR)
nfd = os.open(native_file, os.O_RDWR)
try:
os.lseek(mfd, 0, os.SEEK_END)
os.lseek(nfd, 0, os.SEEK_END)
os.write(mfd, b"-appended")
os.write(nfd, b"-appended")
touch(merge_file, b"abc")
touch(native_file, b"abc")
os.makedirs(merge_dir, exist_ok=True)
os.makedirs(native_dir, exist_ok=True)
err = compare_calls("fsync file", lambda: os.fsync(mfd), lambda: os.fsync(nfd))
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
mfd = os.open(merge_file, os.O_RDWR)
nfd = os.open(native_file, os.O_RDWR)
try:
os.lseek(mfd, 0, os.SEEK_END)
os.lseek(nfd, 0, os.SEEK_END)
os.write(mfd, b"-appended")
os.write(nfd, b"-appended")
mdirfd = os.open(merge_dir, os.O_RDONLY | os.O_DIRECTORY)
ndirfd = os.open(native_dir, os.O_RDONLY | os.O_DIRECTORY)
try:
err = compare_calls("fsync dir", lambda: os.fsync(mdirfd), lambda: os.fsync(ndirfd))
if err:
return fail(err)
finally:
os.close(mdirfd)
os.close(ndirfd)
err = compare_calls("fsync file", lambda: os.fsync(mfd), lambda: os.fsync(nfd))
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
bad_m = os.open(merge_file, os.O_RDONLY)
bad_n = os.open(native_file, os.O_RDONLY)
os.close(bad_m)
os.close(bad_n)
err = compare_calls("fsync EBADF", lambda: os.fsync(bad_m), lambda: os.fsync(bad_n))
if err:
return fail(err)
mdirfd = os.open(merge_dir, os.O_RDONLY | os.O_DIRECTORY)
ndirfd = os.open(native_dir, os.O_RDONLY | os.O_DIRECTORY)
try:
err = compare_calls("fsync dir", lambda: os.fsync(mdirfd), lambda: os.fsync(ndirfd))
if err:
return fail(err)
finally:
os.close(mdirfd)
os.close(ndirfd)
with open(merge_file, "rb") as mf, open(native_file, "rb") as nf:
mdata = mf.read()
ndata = nf.read()
if mdata != ndata:
return fail(f"flush content mismatch mergerfs={mdata!r} native={ndata!r}")
bad_m = os.open(merge_file, os.O_RDONLY)
bad_n = os.open(native_file, os.O_RDONLY)
os.close(bad_m)
os.close(bad_n)
err = compare_calls("fsync EBADF", lambda: os.fsync(bad_m), lambda: os.fsync(bad_n))
if err:
return fail(err)
# Close-to-flush parity check: both paths should preserve appended payload.
with open(merge_file, "rb") as mf, open(native_file, "rb") as nf:
mdata = mf.read()
ndata = nf.read()
if mdata != ndata:
return fail(f"flush content mismatch mergerfs={mdata!r} native={ndata!r}")
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+64 -65
View File
@@ -10,93 +10,92 @@ from posix_parity import cleanup_paths
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import should_compare_inode
from posix_parity import temp_dir
from posix_parity import touch
def st_cmp(lhs, rhs):
return (
def st_cmp(lhs, rhs, compare_ino=False):
base = (
stat.S_IFMT(lhs.st_mode) == stat.S_IFMT(rhs.st_mode)
and stat.S_IMODE(lhs.st_mode) == stat.S_IMODE(rhs.st_mode)
and lhs.st_size == rhs.st_size
and lhs.st_nlink == rhs.st_nlink
)
def st_cmp_with_inode(lhs, rhs):
return st_cmp(lhs, rhs) and lhs.st_ino == rhs.st_ino
if stat.S_ISDIR(lhs.st_mode) or stat.S_ISDIR(rhs.st_mode):
return base and (not compare_ino or lhs.st_ino == rhs.st_ino)
return base and lhs.st_size == rhs.st_size and (not compare_ino or lhs.st_ino == rhs.st_ino)
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_getattr_fgetattr <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
compare_ino = should_compare_inode(mount)
mount = sys.argv[1]
stcmp = st_cmp_with_inode if should_compare_inode(mount) else st_cmp
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
cleanup_paths([merge_file, merge_dir, merge_notdir])
cleanup_paths([merge_file, merge_dir, merge_notdir])
touch(merge_file, b"abc", 0o640)
touch(native_file, b"abc", 0o640)
os.makedirs(merge_dir, exist_ok=True)
os.makedirs(native_dir, exist_ok=True)
touch(merge_notdir, b"x", 0o644)
touch(native_notdir, b"x", 0o644)
touch(merge_file, b"abc", 0o640)
touch(native_file, b"abc", 0o640)
os.makedirs(merge_dir, exist_ok=True)
os.makedirs(native_dir, exist_ok=True)
touch(merge_notdir, b"x", 0o644)
touch(native_notdir, b"x", 0o644)
err = compare_calls(
"lstat regular", lambda: os.lstat(merge_file), lambda: os.lstat(native_file), lambda l, r: st_cmp(l, r, compare_ino)
)
if err:
return fail(err)
err = compare_calls(
"lstat regular", lambda: os.lstat(merge_file), lambda: os.lstat(native_file), stcmp
)
if err:
return fail(err)
err = compare_calls(
"lstat directory", lambda: os.lstat(merge_dir), lambda: os.lstat(native_dir), lambda l, r: st_cmp(l, r, compare_ino)
)
if err:
return fail(err)
err = compare_calls(
"lstat directory", lambda: os.lstat(merge_dir), lambda: os.lstat(native_dir), stcmp
)
if err:
return fail(err)
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
try:
err = compare_calls("fstat open fd", lambda: os.fstat(mfd), lambda: os.fstat(nfd), lambda l, r: st_cmp(l, r, compare_ino))
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
try:
err = compare_calls("fstat open fd", lambda: os.fstat(mfd), lambda: os.fstat(nfd), stcmp)
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
err = compare_calls("lstat ENOENT", lambda: os.lstat(merge_missing), lambda: os.lstat(native_missing))
if err:
return fail(err)
err = compare_calls("lstat ENOENT", lambda: os.lstat(merge_missing), lambda: os.lstat(native_missing))
if err:
return fail(err)
err = compare_calls(
"lstat ENOTDIR",
lambda: os.lstat(join(merge_notdir, "child")),
lambda: os.lstat(join(native_notdir, "child")),
)
if err:
return fail(err)
err = compare_calls(
"lstat ENOTDIR",
lambda: os.lstat(join(merge_notdir, "child")),
lambda: os.lstat(join(native_notdir, "child")),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+37 -37
View File
@@ -11,6 +11,7 @@ from posix_parity import cleanup_dir
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
@@ -22,47 +23,46 @@ def fionread(fd):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_ioctl <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
touch(merge_file, b"hello-world")
touch(native_file, b"hello-world")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
try:
err = compare_calls("ioctl FIONREAD", lambda: fionread(mfd), lambda: fionread(nfd), lambda a, b: a == b)
if err:
return fail(err)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
touch(merge_file, b"hello-world")
touch(native_file, b"hello-world")
os.lseek(mfd, 5, os.SEEK_SET)
os.lseek(nfd, 5, os.SEEK_SET)
err = compare_calls(
"ioctl FIONREAD after seek",
lambda: fionread(mfd),
lambda: fionread(nfd),
lambda a, b: a == b,
)
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
try:
err = compare_calls("ioctl FIONREAD", lambda: fionread(mfd), lambda: fionread(nfd), lambda a, b: a == b)
if err:
return fail(err)
os.lseek(mfd, 5, os.SEEK_SET)
os.lseek(nfd, 5, os.SEEK_SET)
err = compare_calls(
"ioctl FIONREAD after seek",
lambda: fionread(mfd),
lambda: fionread(nfd),
lambda a, b: a == b,
)
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+49 -49
View File
@@ -9,67 +9,67 @@ from posix_parity import cleanup_paths
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_link_symlink <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_src = join(merge_base, "src")
native_src = join(native_base, "src")
merge_dst = join(merge_base, "dst")
native_dst = join(native_base, "dst")
merge_slnk = join(merge_base, "symlink")
native_slnk = join(native_base, "symlink")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
cleanup_paths([merge_src, merge_dst, merge_slnk])
touch(merge_src, b"src", 0o644)
touch(native_src, b"src", 0o644)
merge_src = join(merge_base, "src")
native_src = join(native_base, "src")
merge_dst = join(merge_base, "dst")
native_dst = join(native_base, "dst")
merge_slnk = join(merge_base, "symlink")
native_slnk = join(native_base, "symlink")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
err = compare_calls("link success", lambda: os.link(merge_src, merge_dst), lambda: os.link(native_src, native_dst))
if err:
return fail(err)
cleanup_paths([merge_src, merge_dst, merge_slnk])
touch(merge_src, b"src", 0o644)
touch(native_src, b"src", 0o644)
err = compare_calls(
"link ENOENT source",
lambda: os.link(merge_missing, join(merge_base, "dst2")),
lambda: os.link(native_missing, join(native_base, "dst2")),
)
if err:
return fail(err)
err = compare_calls("link success", lambda: os.link(merge_src, merge_dst), lambda: os.link(native_src, native_dst))
if err:
return fail(err)
err = compare_calls(
"symlink success",
lambda: os.symlink("src", merge_slnk),
lambda: os.symlink("src", native_slnk),
)
if err:
return fail(err)
err = compare_calls(
"link ENOENT source",
lambda: os.link(merge_missing, join(merge_base, "dst2")),
lambda: os.link(native_missing, join(native_base, "dst2")),
)
if err:
return fail(err)
err = compare_calls(
"symlink EEXIST",
lambda: os.symlink("src", merge_slnk),
lambda: os.symlink("src", native_slnk),
)
if err:
return fail(err)
err = compare_calls(
"symlink success",
lambda: os.symlink("src", merge_slnk),
lambda: os.symlink("src", native_slnk),
)
if err:
return fail(err)
err = compare_calls(
"symlink EEXIST",
lambda: os.symlink("src", merge_slnk),
lambda: os.symlink("src", native_slnk),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+73 -75
View File
@@ -12,12 +12,12 @@ from posix_parity import cleanup_dir
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def pack_flock(l_type, l_whence=0, l_start=0, l_len=0, l_pid=0):
# struct flock (common glibc x86_64 layout)
return struct.pack("hhqqi", l_type, l_whence, l_start, l_len, l_pid)
@@ -51,91 +51,89 @@ def main():
print(try_process_lock(sys.argv[2]))
return 0
if len(sys.argv) != 2:
print("usage: TEST_posix_locking <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
touch(merge_file, b"x")
touch(native_file, b"x")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mfd1 = os.open(merge_file, os.O_RDWR)
mfd2 = os.open(merge_file, os.O_RDWR)
nfd1 = os.open(native_file, os.O_RDWR)
nfd2 = os.open(native_file, os.O_RDWR)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
touch(merge_file, b"x")
touch(native_file, b"x")
try:
err = compare_calls(
"flock EX nonblock",
lambda: fcntl.flock(mfd1, fcntl.LOCK_EX | fcntl.LOCK_NB),
lambda: fcntl.flock(nfd1, fcntl.LOCK_EX | fcntl.LOCK_NB),
)
if err:
return fail(err)
mfd1 = os.open(merge_file, os.O_RDWR)
mfd2 = os.open(merge_file, os.O_RDWR)
nfd1 = os.open(native_file, os.O_RDWR)
nfd2 = os.open(native_file, os.O_RDWR)
err = compare_calls(
"flock SH nonblock second fd",
lambda: fcntl.flock(mfd2, fcntl.LOCK_SH | fcntl.LOCK_NB),
lambda: fcntl.flock(nfd2, fcntl.LOCK_SH | fcntl.LOCK_NB),
)
if err:
return fail(err)
try:
err = compare_calls(
"flock EX nonblock",
lambda: fcntl.flock(mfd1, fcntl.LOCK_EX | fcntl.LOCK_NB),
lambda: fcntl.flock(nfd1, fcntl.LOCK_EX | fcntl.LOCK_NB),
)
if err:
return fail(err)
err = compare_calls("flock unlock", lambda: fcntl.flock(mfd1, fcntl.LOCK_UN), lambda: fcntl.flock(nfd1, fcntl.LOCK_UN))
if err:
return fail(err)
err = compare_calls("flock unlock second fd", lambda: fcntl.flock(mfd2, fcntl.LOCK_UN), lambda: fcntl.flock(nfd2, fcntl.LOCK_UN))
if err:
return fail(err)
err = compare_calls(
"flock SH nonblock second fd",
lambda: fcntl.flock(mfd2, fcntl.LOCK_SH | fcntl.LOCK_NB),
lambda: fcntl.flock(nfd2, fcntl.LOCK_SH | fcntl.LOCK_NB),
)
if err:
return fail(err)
wrlk = pack_flock(fcntl.F_WRLCK)
unlck = pack_flock(fcntl.F_UNLCK)
err = compare_calls("flock unlock", lambda: fcntl.flock(mfd1, fcntl.LOCK_UN), lambda: fcntl.flock(nfd1, fcntl.LOCK_UN))
if err:
return fail(err)
err = compare_calls("flock unlock second fd", lambda: fcntl.flock(mfd2, fcntl.LOCK_UN), lambda: fcntl.flock(nfd2, fcntl.LOCK_UN))
if err:
return fail(err)
err = compare_calls("fcntl F_SETLK write lock", lambda: fcntl.fcntl(mfd1, fcntl.F_SETLK, wrlk), lambda: fcntl.fcntl(nfd1, fcntl.F_SETLK, wrlk))
if err:
return fail(err)
# POSIX record lock via fcntl
wrlk = pack_flock(fcntl.F_WRLCK)
unlck = pack_flock(fcntl.F_UNLCK)
m_lock_errno = child_lock_errno(merge_file)
n_lock_errno = child_lock_errno(native_file)
if m_lock_errno != n_lock_errno:
return fail(f"fcntl F_SETLK child contention errno mismatch mergerfs={m_lock_errno} native={n_lock_errno}")
if m_lock_errno not in (errno.EACCES, errno.EAGAIN):
return fail(f"fcntl F_SETLK child contention expected lock conflict got errno={m_lock_errno}")
err = compare_calls("fcntl F_SETLK write lock", lambda: fcntl.fcntl(mfd1, fcntl.F_SETLK, wrlk), lambda: fcntl.fcntl(nfd1, fcntl.F_SETLK, wrlk))
if err:
return fail(err)
err = compare_calls("fcntl F_SETLK unlock", lambda: fcntl.fcntl(mfd1, fcntl.F_SETLK, unlck), lambda: fcntl.fcntl(nfd1, fcntl.F_SETLK, unlck))
if err:
return fail(err)
m_lock_errno = child_lock_errno(merge_file)
n_lock_errno = child_lock_errno(native_file)
if m_lock_errno != n_lock_errno:
return fail(f"fcntl F_SETLK child contention errno mismatch mergerfs={m_lock_errno} native={n_lock_errno}")
if m_lock_errno not in (errno.EACCES, errno.EAGAIN):
return fail(f"fcntl F_SETLK child contention expected lock conflict got errno={m_lock_errno}")
bad_m = os.open(merge_file, os.O_RDONLY)
bad_n = os.open(native_file, os.O_RDONLY)
os.close(bad_m)
os.close(bad_n)
err = compare_calls(
"fcntl EBADF",
lambda: fcntl.fcntl(bad_m, fcntl.F_GETFD),
lambda: fcntl.fcntl(bad_n, fcntl.F_GETFD),
)
if err:
return fail(err)
finally:
os.close(mfd1)
os.close(mfd2)
os.close(nfd1)
os.close(nfd2)
err = compare_calls("fcntl F_SETLK unlock", lambda: fcntl.fcntl(mfd1, fcntl.F_SETLK, unlck), lambda: fcntl.fcntl(nfd1, fcntl.F_SETLK, unlck))
if err:
return fail(err)
bad_m = os.open(merge_file, os.O_RDONLY)
bad_n = os.open(native_file, os.O_RDONLY)
os.close(bad_m)
os.close(bad_n)
err = compare_calls(
"fcntl EBADF",
lambda: fcntl.fcntl(bad_m, fcntl.F_GETFD),
lambda: fcntl.fcntl(bad_n, fcntl.F_GETFD),
)
if err:
return fail(err)
finally:
os.close(mfd1)
os.close(mfd2)
os.close(nfd1)
os.close(nfd2)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+49 -49
View File
@@ -9,69 +9,69 @@ from posix_parity import cleanup_paths
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_mkdir_rmdir <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
cleanup_paths([merge_dir, merge_notdir])
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
err = compare_calls("mkdir success", lambda: os.mkdir(merge_dir, 0o755), lambda: os.mkdir(native_dir, 0o755))
if err:
return fail(err)
cleanup_paths([merge_dir, merge_notdir])
err = compare_calls("mkdir EEXIST", lambda: os.mkdir(merge_dir, 0o755), lambda: os.mkdir(native_dir, 0o755))
if err:
return fail(err)
err = compare_calls("mkdir success", lambda: os.mkdir(merge_dir, 0o755), lambda: os.mkdir(native_dir, 0o755))
if err:
return fail(err)
touch(merge_notdir, b"x")
touch(native_notdir, b"x")
err = compare_calls(
"mkdir ENOTDIR",
lambda: os.mkdir(join(merge_notdir, "child"), 0o755),
lambda: os.mkdir(join(native_notdir, "child"), 0o755),
)
if err:
return fail(err)
err = compare_calls("mkdir EEXIST", lambda: os.mkdir(merge_dir, 0o755), lambda: os.mkdir(native_dir, 0o755))
if err:
return fail(err)
err = compare_calls("rmdir success", lambda: os.rmdir(merge_dir), lambda: os.rmdir(native_dir))
if err:
return fail(err)
touch(merge_notdir, b"x")
touch(native_notdir, b"x")
err = compare_calls(
"mkdir ENOTDIR",
lambda: os.mkdir(join(merge_notdir, "child"), 0o755),
lambda: os.mkdir(join(native_notdir, "child"), 0o755),
)
if err:
return fail(err)
err = compare_calls("rmdir ENOENT", lambda: os.rmdir(merge_dir), lambda: os.rmdir(native_dir))
if err:
return fail(err)
err = compare_calls("rmdir success", lambda: os.rmdir(merge_dir), lambda: os.rmdir(native_dir))
if err:
return fail(err)
touch(join(merge_base, "nonempty/file"), b"x")
touch(join(native_base, "nonempty/file"), b"x")
err = compare_calls(
"rmdir ENOTEMPTY",
lambda: os.rmdir(join(merge_base, "nonempty")),
lambda: os.rmdir(join(native_base, "nonempty")),
)
if err:
return fail(err)
err = compare_calls("rmdir ENOENT", lambda: os.rmdir(merge_dir), lambda: os.rmdir(native_dir))
if err:
return fail(err)
touch(join(merge_base, "nonempty/file"), b"x")
touch(join(native_base, "nonempty/file"), b"x")
err = compare_calls(
"rmdir ENOTEMPTY",
lambda: os.rmdir(join(merge_base, "nonempty")),
lambda: os.rmdir(join(native_base, "nonempty")),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+60 -60
View File
@@ -9,79 +9,79 @@ from posix_parity import cleanup_paths
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_open_read_write <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
cleanup_paths([merge_file, merge_notdir])
touch(merge_file, b"123456", 0o644)
touch(native_file, b"123456", 0o644)
os.makedirs(merge_dir, exist_ok=True)
os.makedirs(native_dir, exist_ok=True)
touch(merge_notdir, b"x", 0o644)
touch(native_notdir, b"x", 0o644)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
err = compare_calls("open ENOENT", lambda: os.open(merge_missing, os.O_RDONLY), lambda: os.open(native_missing, os.O_RDONLY))
if err:
return fail(err)
cleanup_paths([merge_file, merge_notdir])
touch(merge_file, b"123456", 0o644)
touch(native_file, b"123456", 0o644)
os.makedirs(merge_dir, exist_ok=True)
os.makedirs(native_dir, exist_ok=True)
touch(merge_notdir, b"x", 0o644)
touch(native_notdir, b"x", 0o644)
err = compare_calls("open EISDIR", lambda: os.open(merge_dir, os.O_WRONLY), lambda: os.open(native_dir, os.O_WRONLY))
if err:
return fail(err)
err = compare_calls("open ENOENT", lambda: os.open(merge_missing, os.O_RDONLY), lambda: os.open(native_missing, os.O_RDONLY))
if err:
return fail(err)
err = compare_calls(
"open ENOTDIR",
lambda: os.open(join(merge_notdir, "child"), os.O_RDONLY),
lambda: os.open(join(native_notdir, "child"), os.O_RDONLY),
)
if err:
return fail(err)
err = compare_calls("open EISDIR", lambda: os.open(merge_dir, os.O_WRONLY), lambda: os.open(native_dir, os.O_WRONLY))
if err:
return fail(err)
mfd = os.open(merge_file, os.O_RDWR)
nfd = os.open(native_file, os.O_RDWR)
try:
os.lseek(mfd, 0, os.SEEK_SET)
os.lseek(nfd, 0, os.SEEK_SET)
m_data = os.read(mfd, 6)
n_data = os.read(nfd, 6)
if m_data != n_data:
return fail(f"read parity mismatch mergerfs={m_data!r} native={n_data!r}")
err = compare_calls(
"open ENOTDIR",
lambda: os.open(join(merge_notdir, "child"), os.O_RDONLY),
lambda: os.open(join(native_notdir, "child"), os.O_RDONLY),
)
if err:
return fail(err)
os.lseek(mfd, 0, os.SEEK_SET)
os.lseek(nfd, 0, os.SEEK_SET)
mw = os.write(mfd, b"abc")
nw = os.write(nfd, b"abc")
if mw != nw:
return fail(f"write count mismatch mergerfs={mw} native={nw}")
finally:
os.close(mfd)
os.close(nfd)
mfd = os.open(merge_file, os.O_RDWR)
nfd = os.open(native_file, os.O_RDWR)
try:
os.lseek(mfd, 0, os.SEEK_SET)
os.lseek(nfd, 0, os.SEEK_SET)
m_data = os.read(mfd, 6)
n_data = os.read(nfd, 6)
if m_data != n_data:
return fail(f"read parity mismatch mergerfs={m_data!r} native={n_data!r}")
os.lseek(mfd, 0, os.SEEK_SET)
os.lseek(nfd, 0, os.SEEK_SET)
mw = os.write(mfd, b"abc")
nw = os.write(nfd, b"abc")
if mw != nw:
return fail(f"write count mismatch mergerfs={mw} native={nw}")
finally:
os.close(mfd)
os.close(nfd)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+30 -31
View File
@@ -8,6 +8,7 @@ import tempfile
from posix_parity import cleanup_dir
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
@@ -23,41 +24,39 @@ def poll_mask(fd, timeout_ms=25):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_poll <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
touch(merge_file, b"abcdef")
touch(native_file, b"abcdef")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
try:
mmask = poll_mask(mfd)
nmask = poll_mask(nfd)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
touch(merge_file, b"abcdef")
touch(native_file, b"abcdef")
if bool(mmask & select.POLLIN) != bool(nmask & select.POLLIN):
return fail(f"poll POLLIN mismatch mergerfs_mask=0x{mmask:x} native_mask=0x{nmask:x}")
if bool(mmask & select.POLLOUT) != bool(nmask & select.POLLOUT):
return fail(f"poll POLLOUT mismatch mergerfs_mask=0x{mmask:x} native_mask=0x{nmask:x}")
finally:
os.close(mfd)
os.close(nfd)
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
try:
mmask = poll_mask(mfd)
nmask = poll_mask(nfd)
# For regular files POLLIN and POLLOUT are typically ready.
if bool(mmask & select.POLLIN) != bool(nmask & select.POLLIN):
return fail(f"poll POLLIN mismatch mergerfs_mask=0x{mmask:x} native_mask=0x{nmask:x}")
if bool(mmask & select.POLLOUT) != bool(nmask & select.POLLOUT):
return fail(f"poll POLLOUT mismatch mergerfs_mask=0x{mmask:x} native_mask=0x{nmask:x}")
finally:
os.close(mfd)
os.close(nfd)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+53 -53
View File
@@ -9,6 +9,7 @@ from posix_parity import cleanup_dir
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
@@ -111,68 +112,67 @@ def deleted_visibility_signature(dirpath, victim_path):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_readdir <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
for idx in range(8):
touch(join(merge_dir, f"f{idx}"), b"x")
touch(join(native_dir, f"f{idx}"), b"x")
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
touch(merge_notdir, b"x")
touch(native_notdir, b"x")
for idx in range(8):
touch(join(merge_dir, f"f{idx}"), b"x")
touch(join(native_dir, f"f{idx}"), b"x")
err = compare_calls(
"opendir ENOENT",
lambda: opendir_or_raise(join(merge_base, "missing")),
lambda: opendir_or_raise(join(native_base, "missing")),
)
if err:
return fail(err)
touch(merge_notdir, b"x")
touch(native_notdir, b"x")
err = compare_calls(
"opendir ENOTDIR",
lambda: opendir_or_raise(join(merge_notdir, "child")),
lambda: opendir_or_raise(join(native_notdir, "child")),
)
if err:
return fail(err)
err = compare_calls(
"opendir ENOENT",
lambda: opendir_or_raise(join(merge_base, "missing")),
lambda: opendir_or_raise(join(native_base, "missing")),
)
if err:
return fail(err)
merge_names = names_set(merge_dir)
native_names = names_set(native_dir)
if merge_names != native_names:
return fail(f"readdir names mismatch mergerfs={sorted(merge_names)} native={sorted(native_names)}")
err = compare_calls(
"opendir ENOTDIR",
lambda: opendir_or_raise(join(merge_notdir, "child")),
lambda: opendir_or_raise(join(native_notdir, "child")),
)
if err:
return fail(err)
merge_consistent = seek_tell_consistent(merge_dir)
native_consistent = seek_tell_consistent(native_dir)
if merge_consistent != native_consistent:
return fail(
f"seekdir/telldir consistency mismatch mergerfs={merge_consistent} native={native_consistent}"
)
merge_names = names_set(merge_dir)
native_names = names_set(native_dir)
if merge_names != native_names:
return fail(f"readdir names mismatch mergerfs={sorted(merge_names)} native={sorted(native_names)}")
touch(join(merge_dir, "victim"), b"x")
touch(join(native_dir, "victim"), b"x")
merge_sig = deleted_visibility_signature(merge_dir, join(merge_dir, "victim"))
native_sig = deleted_visibility_signature(native_dir, join(native_dir, "victim"))
if merge_sig != native_sig:
return fail(f"deleted-entry visibility mismatch mergerfs={merge_sig} native={native_sig}")
merge_consistent = seek_tell_consistent(merge_dir)
native_consistent = seek_tell_consistent(native_dir)
if merge_consistent != native_consistent:
return fail(
f"seekdir/telldir consistency mismatch mergerfs={merge_consistent} native={native_consistent}"
)
touch(join(merge_dir, "victim"), b"x")
touch(join(native_dir, "victim"), b"x")
merge_sig = deleted_visibility_signature(merge_dir, join(merge_dir, "victim"))
native_sig = deleted_visibility_signature(native_dir, join(native_dir, "victim"))
if merge_sig != native_sig:
return fail(f"deleted-entry visibility mismatch mergerfs={merge_sig} native={native_sig}")
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+44 -46
View File
@@ -8,6 +8,7 @@ from posix_parity import cleanup_dir
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_get_option
from posix_parity import mergerfs_mount
from posix_parity import mergerfs_set_option
from posix_parity import temp_dir
from posix_parity import touch
@@ -24,54 +25,51 @@ def list_and_stat(dirpath):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_readdir_plus <mountpoint>", file=sys.stderr)
return 1
mount = sys.argv[1]
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
for i in range(40):
touch(join(merge_dir, f"f{i:03d}"), bytes(str(i), "ascii"))
touch(join(native_dir, f"f{i:03d}"), bytes(str(i), "ascii"))
# Exercise mapping from readdir entries to immediate getattr calls.
m_map = list_and_stat(merge_dir)
n_map = list_and_stat(native_dir)
if m_map != n_map:
return fail("readdir+getattr map mismatch between mergerfs and native")
# Optionally toggle readdir policy and ensure semantic output stays stable.
try:
orig = mergerfs_get_option(mount, "func.readdir")
except (PermissionError, FileNotFoundError, OSError):
return 0
try:
for mode in ("seq", "cosr:2:2", "cor:2:2"):
mergerfs_set_option(mount, "func.readdir", mode)
got = list_and_stat(merge_dir)
if got != m_map:
return fail(f"func.readdir={mode} changed readdir+getattr semantic output")
except (PermissionError, FileNotFoundError, OSError):
return 0
finally:
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
mergerfs_set_option(mount, "func.readdir", orig)
except OSError:
pass
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
return 0
finally:
cleanup_dir(merge_base)
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
for i in range(40):
touch(join(merge_dir, f"f{i:03d}"), bytes(str(i), "ascii"))
touch(join(native_dir, f"f{i:03d}"), bytes(str(i), "ascii"))
m_map = list_and_stat(merge_dir)
n_map = list_and_stat(native_dir)
if m_map != n_map:
return fail("readdir+getattr map mismatch between mergerfs and native")
try:
orig = mergerfs_get_option(mount, "func.readdir")
except (PermissionError, FileNotFoundError, OSError):
return 0
try:
for mode in ("seq", "cosr:2:2", "cor:2:2"):
mergerfs_set_option(mount, "func.readdir", mode)
got = list_and_stat(merge_dir)
if got != m_map:
return fail(f"func.readdir={mode} changed readdir+getattr semantic output")
except (PermissionError, FileNotFoundError, OSError):
return 0
finally:
try:
mergerfs_set_option(mount, "func.readdir", orig)
except OSError:
pass
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+59 -61
View File
@@ -7,81 +7,79 @@ import tempfile
from posix_parity import cleanup_dir
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_release <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
touch(merge_file, b"abc")
touch(native_file, b"abc")
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
mfd1 = os.open(merge_file, os.O_RDWR)
nfd1 = os.open(native_file, os.O_RDWR)
mfd2 = os.dup(mfd1)
nfd2 = os.dup(nfd1)
touch(merge_file, b"abc")
touch(native_file, b"abc")
try:
os.close(mfd1)
os.close(nfd1)
# Open + dup so one close does not release the underlying open-file state.
mfd1 = os.open(merge_file, os.O_RDWR)
nfd1 = os.open(native_file, os.O_RDWR)
mfd2 = os.dup(mfd1)
nfd2 = os.dup(nfd1)
mw = os.write(mfd2, b"X")
nw = os.write(nfd2, b"X")
if mw != nw:
return fail(f"release dup write mismatch mergerfs={mw} native={nw}")
try:
os.close(mfd1)
os.close(nfd1)
os.lseek(mfd2, 0, os.SEEK_SET)
os.lseek(nfd2, 0, os.SEEK_SET)
mr = os.read(mfd2, 4)
nr = os.read(nfd2, 4)
if mr != nr:
return fail(f"release dup read mismatch mergerfs={mr!r} native={nr!r}")
finally:
os.close(mfd2)
os.close(nfd2)
mw = os.write(mfd2, b"X")
nw = os.write(nfd2, b"X")
if mw != nw:
return fail(f"release dup write mismatch mergerfs={mw} native={nw}")
with open(merge_file, "rb") as mf, open(native_file, "rb") as nf:
md = mf.read()
nd = nf.read()
if md != nd:
return fail(f"release final data mismatch mergerfs={md!r} native={nd!r}")
os.lseek(mfd2, 0, os.SEEK_SET)
os.lseek(nfd2, 0, os.SEEK_SET)
mr = os.read(mfd2, 4)
nr = os.read(nfd2, 4)
if mr != nr:
return fail(f"release dup read mismatch mergerfs={mr!r} native={nr!r}")
finally:
os.close(mfd2)
os.close(nfd2)
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
os.close(mfd)
os.close(nfd)
try:
os.close(mfd)
m_err = 0
except OSError as exc:
m_err = exc.errno
try:
os.close(nfd)
n_err = 0
except OSError as exc:
n_err = exc.errno
if m_err != n_err:
return fail(f"release EBADF mismatch mergerfs_errno={m_err} native_errno={n_err}")
with open(merge_file, "rb") as mf, open(native_file, "rb") as nf:
md = mf.read()
nd = nf.read()
if md != nd:
return fail(f"release final data mismatch mergerfs={md!r} native={nd!r}")
# Closing same fd twice should map to EBADF on both.
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
os.close(mfd)
os.close(nfd)
try:
os.close(mfd)
m_err = 0
except OSError as exc:
m_err = exc.errno
try:
os.close(nfd)
n_err = 0
except OSError as exc:
n_err = exc.errno
if m_err != n_err:
return fail(f"release EBADF mismatch mergerfs_errno={m_err} native_errno={n_err}")
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+22 -22
View File
@@ -8,6 +8,7 @@ import tempfile
from posix_parity import cleanup_dir
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
@@ -49,31 +50,30 @@ def close_and_dirfd_errno(path):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_releasedir <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
touch(join(merge_dir, "a"), b"x")
touch(join(native_dir, "a"), b"x")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
m_err = close_and_dirfd_errno(merge_dir)
n_err = close_and_dirfd_errno(native_dir)
if m_err != n_err:
return fail(f"releasedir EBADF parity mismatch mergerfs_errno={m_err} native_errno={n_err}")
merge_dir = join(merge_base, "dir")
native_dir = join(native_base, "dir")
touch(join(merge_dir, "a"), b"x")
touch(join(native_dir, "a"), b"x")
m_err = close_and_dirfd_errno(merge_dir)
n_err = close_and_dirfd_errno(native_dir)
if m_err != n_err:
return fail(f"releasedir EBADF parity mismatch mergerfs_errno={m_err} native_errno={n_err}")
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+29 -29
View File
@@ -8,6 +8,7 @@ from posix_parity import cleanup_dir
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
@@ -22,38 +23,37 @@ def statvfs_cmp(lhs, rhs):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_statfs <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
err = compare_calls(
"statvfs mount parity",
lambda: os.statvfs(mount),
lambda: os.statvfs(native),
statvfs_cmp,
)
if err:
return fail(err)
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
err = compare_calls(
"statvfs ENOENT",
lambda: os.statvfs(join(merge_base, "missing")),
lambda: os.statvfs(join(native_base, "missing")),
)
if err:
return fail(err)
err = compare_calls(
"statvfs mount parity",
lambda: os.statvfs(mount),
lambda: os.statvfs(native),
statvfs_cmp,
)
if err:
return fail(err)
err = compare_calls(
"statvfs ENOENT",
lambda: os.statvfs(join(merge_base, "missing")),
lambda: os.statvfs(join(native_base, "missing")),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+85 -87
View File
@@ -9,6 +9,7 @@ from posix_parity import cleanup_dir
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_fullpath
from posix_parity import mergerfs_mount
from posix_parity import should_compare_inode
from posix_parity import temp_dir
from posix_parity import touch
@@ -172,100 +173,97 @@ def expect_same_errno(name, mcall, ncall):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_statx <mountpoint>", file=sys.stderr)
return 1
mount = sys.argv[1]
stcmp = cmp_statx_basic_with_inode if should_compare_inode(mount) else cmp_statx_basic
merge_base = temp_dir(mount)
try:
merge_file = join(merge_base, "file")
merge_link = join(merge_base, "link")
with mergerfs_mount() as (mount, _):
stcmp = cmp_statx_basic_with_inode if should_compare_inode(mount) else cmp_statx_basic
touch(merge_file, b"hello")
merge_base = temp_dir(mount)
try:
native_file = mergerfs_fullpath(merge_file)
except OSError:
return 0
try:
merge_file = join(merge_base, "file")
merge_link = join(merge_base, "link")
try:
os.unlink(merge_link)
except FileNotFoundError:
pass
os.symlink("file", merge_link)
touch(merge_file, b"hello")
try:
native_link = mergerfs_fullpath(merge_link)
except OSError:
return 0
try:
native_file = mergerfs_fullpath(merge_file)
except OSError:
return 0
try:
os.unlink(merge_link)
except FileNotFoundError:
pass
os.symlink("file", merge_link)
try:
native_link = mergerfs_fullpath(merge_link)
except OSError:
return 0
expected_native_link = os.path.join(os.path.dirname(native_file), "link")
if not os.path.islink(native_link):
if os.path.islink(expected_native_link):
native_link = expected_native_link
else:
return 0
native_missing = os.path.join(os.path.dirname(native_file), "missing")
mst, nst, err = call_pair(
"statx regular",
lambda: statx_call(merge_file),
lambda: statx_call(native_file),
)
if err:
return fail(err)
if not stcmp(mst, nst):
return fail(
"statx basic mismatch\n"
f" mergerfs path: {merge_file}\n"
f" native path: {native_file}\n"
f" mergerfs statx: {statx_summary(mst)}\n"
f" native statx: {statx_summary(nst)}"
)
mst, nst, err = call_pair(
"statx symlink nofollow",
lambda: statx_call(merge_link, flags=AT_SYMLINK_NOFOLLOW),
lambda: statx_call(native_link, flags=AT_SYMLINK_NOFOLLOW),
)
if err:
return fail(err)
if (mst.stx_mode & 0xF000) != (nst.stx_mode & 0xF000):
return fail(
"statx symlink type mismatch\n"
f" mergerfs path: {merge_link}\n"
f" native path: {native_link}\n"
f" mergerfs statx: {statx_summary(mst)}\n"
f" native statx: {statx_summary(nst)}"
)
err = expect_same_errno(
"statx ENOENT",
lambda: statx_call(join(merge_base, "missing")),
lambda: statx_call(native_missing),
)
if err:
return fail(err)
err = expect_same_errno(
"statx ENOTDIR",
lambda: statx_call(join(merge_file, "child")),
lambda: statx_call(join(native_file, "child")),
)
if err:
return fail(err)
# user.mergerfs.fullpath for a symlink may resolve to the target file path
# depending on policy behavior. For AT_SYMLINK_NOFOLLOW comparisons we need
# the underlying symlink path itself.
expected_native_link = os.path.join(os.path.dirname(native_file), "link")
if not os.path.islink(native_link):
if os.path.islink(expected_native_link):
native_link = expected_native_link
else:
return 0
native_missing = os.path.join(os.path.dirname(native_file), "missing")
mst, nst, err = call_pair(
"statx regular",
lambda: statx_call(merge_file),
lambda: statx_call(native_file),
)
if err:
return fail(err)
if not stcmp(mst, nst):
return fail(
"statx basic mismatch\n"
f" mergerfs path: {merge_file}\n"
f" native path: {native_file}\n"
f" mergerfs statx: {statx_summary(mst)}\n"
f" native statx: {statx_summary(nst)}"
)
mst, nst, err = call_pair(
"statx symlink nofollow",
lambda: statx_call(merge_link, flags=AT_SYMLINK_NOFOLLOW),
lambda: statx_call(native_link, flags=AT_SYMLINK_NOFOLLOW),
)
if err:
return fail(err)
if (mst.stx_mode & 0xF000) != (nst.stx_mode & 0xF000):
return fail(
"statx symlink type mismatch\n"
f" mergerfs path: {merge_link}\n"
f" native path: {native_link}\n"
f" mergerfs statx: {statx_summary(mst)}\n"
f" native statx: {statx_summary(nst)}"
)
err = expect_same_errno(
"statx ENOENT",
lambda: statx_call(join(merge_base, "missing")),
lambda: statx_call(native_missing),
)
if err:
return fail(err)
err = expect_same_errno(
"statx ENOTDIR",
lambda: statx_call(join(merge_file, "child")),
lambda: statx_call(join(native_file, "child")),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+30 -30
View File
@@ -9,6 +9,7 @@ from posix_parity import cleanup_dir
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
@@ -28,40 +29,39 @@ def syncfs_fd(fd):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_syncfs <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
touch(merge_file, b"x")
touch(native_file, b"x")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
try:
err = compare_calls("syncfs success", lambda: syncfs_fd(mfd), lambda: syncfs_fd(nfd))
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
touch(merge_file, b"x")
touch(native_file, b"x")
err = compare_calls("syncfs EBADF", lambda: syncfs_fd(-1), lambda: syncfs_fd(-1))
if err:
return fail(err)
mfd = os.open(merge_file, os.O_RDONLY)
nfd = os.open(native_file, os.O_RDONLY)
try:
err = compare_calls("syncfs success", lambda: syncfs_fd(mfd), lambda: syncfs_fd(nfd))
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
err = compare_calls("syncfs EBADF", lambda: syncfs_fd(-1), lambda: syncfs_fd(-1))
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+38 -41
View File
@@ -7,6 +7,7 @@ import tempfile
from posix_parity import cleanup_dir
from posix_parity import fail
from posix_parity import mergerfs_mount
from posix_parity import join
from posix_parity import temp_dir
@@ -25,55 +26,51 @@ def errno_of(call):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_tmpfile <mountpoint>", file=sys.stderr)
return 1
if not hasattr(os, "O_TMPFILE"):
return 0
mount = sys.argv[1]
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
merge_dir = merge_base
native_dir = native_base
merge_dir = merge_base
native_dir = native_base
m_err = errno_of(lambda: open_tmpfile(merge_dir))
n_err = errno_of(lambda: open_tmpfile(native_dir))
if m_err == errno.EOPNOTSUPP:
return 0
if m_err != n_err:
return fail(f"O_TMPFILE support mismatch mergerfs_errno={m_err} native_errno={n_err}")
m_err = errno_of(lambda: open_tmpfile(merge_dir))
n_err = errno_of(lambda: open_tmpfile(native_dir))
# mergerfs does not currently support O_TMPFILE, so EOPNOTSUPP is
# the expected result even when the native filesystem supports it.
if m_err == errno.EOPNOTSUPP:
return 0
if m_err != n_err:
return fail(f"O_TMPFILE support mismatch mergerfs_errno={m_err} native_errno={n_err}")
if m_err in (errno.EOPNOTSUPP, errno.EISDIR, errno.EINVAL, errno.ENOSYS):
return 0
# If unsupported on both, treat as skip-pass.
if m_err in (errno.EOPNOTSUPP, errno.EISDIR, errno.EINVAL, errno.ENOSYS):
return 0
mfd = open_tmpfile(merge_dir)
try:
nfd = open_tmpfile(native_dir)
except OSError:
os.close(mfd)
return fail("O_TMPFILE: native open failed after merge succeeded")
try:
mw = os.write(mfd, b"tmpfile-data")
nw = os.write(nfd, b"tmpfile-data")
if mw != nw:
return fail(f"O_TMPFILE write count mismatch mergerfs={mw} native={nw}")
finally:
os.close(mfd)
os.close(nfd)
mfd = open_tmpfile(merge_dir)
try:
nfd = open_tmpfile(native_dir)
except OSError:
os.close(mfd)
return fail("O_TMPFILE: native open failed after merge succeeded")
try:
mw = os.write(mfd, b"tmpfile-data")
nw = os.write(nfd, b"tmpfile-data")
if mw != nw:
return fail(f"O_TMPFILE write count mismatch mergerfs={mw} native={nw}")
finally:
os.close(mfd)
os.close(nfd)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+39 -39
View File
@@ -9,57 +9,57 @@ from posix_parity import cleanup_paths
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_truncate_ftruncate <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
cleanup_paths([merge_file])
touch(merge_file, b"1234567890")
touch(native_file, b"1234567890")
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
err = compare_calls("truncate shrink", lambda: os.truncate(merge_file, 3), lambda: os.truncate(native_file, 3))
if err:
return fail(err)
cleanup_paths([merge_file])
touch(merge_file, b"1234567890")
touch(native_file, b"1234567890")
err = compare_calls("truncate ENOENT", lambda: os.truncate(merge_missing, 1), lambda: os.truncate(native_missing, 1))
if err:
return fail(err)
err = compare_calls("truncate shrink", lambda: os.truncate(merge_file, 3), lambda: os.truncate(native_file, 3))
if err:
return fail(err)
mfd = os.open(merge_file, os.O_RDWR)
nfd = os.open(native_file, os.O_RDWR)
try:
err = compare_calls("ftruncate grow", lambda: os.ftruncate(mfd, 16), lambda: os.ftruncate(nfd, 16))
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
err = compare_calls("truncate ENOENT", lambda: os.truncate(merge_missing, 1), lambda: os.truncate(native_missing, 1))
if err:
return fail(err)
err = compare_calls("ftruncate EBADF", lambda: os.ftruncate(-1, 4), lambda: os.ftruncate(-1, 4))
if err:
return fail(err)
mfd = os.open(merge_file, os.O_RDWR)
nfd = os.open(native_file, os.O_RDWR)
try:
err = compare_calls("ftruncate grow", lambda: os.ftruncate(mfd, 16), lambda: os.ftruncate(nfd, 16))
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
err = compare_calls("ftruncate EBADF", lambda: os.ftruncate(-1, 4), lambda: os.ftruncate(-1, 4))
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+50 -50
View File
@@ -9,70 +9,70 @@ from posix_parity import cleanup_paths
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_unlink_rename <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_a = join(merge_base, "a")
native_a = join(native_base, "a")
merge_b = join(merge_base, "b")
native_b = join(native_base, "b")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
cleanup_paths([merge_a, merge_b, merge_notdir])
touch(merge_a, b"a")
touch(native_a, b"a")
merge_a = join(merge_base, "a")
native_a = join(native_base, "a")
merge_b = join(merge_base, "b")
native_b = join(native_base, "b")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
err = compare_calls("unlink success", lambda: os.unlink(merge_a), lambda: os.unlink(native_a))
if err:
return fail(err)
cleanup_paths([merge_a, merge_b, merge_notdir])
touch(merge_a, b"a")
touch(native_a, b"a")
err = compare_calls("unlink ENOENT", lambda: os.unlink(merge_missing), lambda: os.unlink(native_missing))
if err:
return fail(err)
err = compare_calls("unlink success", lambda: os.unlink(merge_a), lambda: os.unlink(native_a))
if err:
return fail(err)
touch(merge_a, b"a")
touch(native_a, b"a")
touch(merge_b, b"b")
touch(native_b, b"b")
err = compare_calls("unlink ENOENT", lambda: os.unlink(merge_missing), lambda: os.unlink(native_missing))
if err:
return fail(err)
err = compare_calls("rename overwrite", lambda: os.rename(merge_a, merge_b), lambda: os.rename(native_a, native_b))
if err:
return fail(err)
touch(merge_a, b"a")
touch(native_a, b"a")
touch(merge_b, b"b")
touch(native_b, b"b")
err = compare_calls("rename ENOENT", lambda: os.rename(merge_missing, merge_a), lambda: os.rename(native_missing, native_a))
if err:
return fail(err)
err = compare_calls("rename overwrite", lambda: os.rename(merge_a, merge_b), lambda: os.rename(native_a, native_b))
if err:
return fail(err)
touch(merge_notdir, b"x")
touch(native_notdir, b"x")
err = compare_calls(
"rename ENOTDIR",
lambda: os.rename(join(merge_notdir, "child"), merge_a),
lambda: os.rename(join(native_notdir, "child"), native_a),
)
if err:
return fail(err)
err = compare_calls("rename ENOENT", lambda: os.rename(merge_missing, merge_a), lambda: os.rename(native_missing, native_a))
if err:
return fail(err)
touch(merge_notdir, b"x")
touch(native_notdir, b"x")
err = compare_calls(
"rename ENOTDIR",
lambda: os.rename(join(merge_notdir, "child"), merge_a),
lambda: os.rename(join(native_notdir, "child"), native_a),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+57 -57
View File
@@ -10,76 +10,76 @@ from posix_parity import cleanup_paths
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_utimens <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
cleanup_paths([merge_file])
touch(merge_file, b"x")
touch(native_file, b"x")
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
now = time.time()
times = (now - 1000, now - 500)
cleanup_paths([merge_file])
touch(merge_file, b"x")
touch(native_file, b"x")
err = compare_calls(
"utime path success",
lambda: os.utime(merge_file, times=times),
lambda: os.utime(native_file, times=times),
)
if err:
return fail(err)
now = time.time()
times = (now - 1000, now - 500)
mfd = os.open(merge_file, os.O_RDWR)
nfd = os.open(native_file, os.O_RDWR)
try:
err = compare_calls(
"utime fd success",
lambda: os.utime(mfd, times=times),
lambda: os.utime(nfd, times=times),
)
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
err = compare_calls(
"utime path success",
lambda: os.utime(merge_file, times=times),
lambda: os.utime(native_file, times=times),
)
if err:
return fail(err)
err = compare_calls(
"utime ENOENT",
lambda: os.utime(merge_missing, times=times),
lambda: os.utime(native_missing, times=times),
)
if err:
return fail(err)
mfd = os.open(merge_file, os.O_RDWR)
nfd = os.open(native_file, os.O_RDWR)
try:
err = compare_calls(
"utime fd success",
lambda: os.utime(mfd, times=times),
lambda: os.utime(nfd, times=times),
)
if err:
return fail(err)
finally:
os.close(mfd)
os.close(nfd)
err = compare_calls(
"utime EBADF",
lambda: os.utime(-1, times=times),
lambda: os.utime(-1, times=times),
)
if err:
return fail(err)
err = compare_calls(
"utime ENOENT",
lambda: os.utime(merge_missing, times=times),
lambda: os.utime(native_missing, times=times),
)
if err:
return fail(err)
err = compare_calls(
"utime EBADF",
lambda: os.utime(-1, times=times),
lambda: os.utime(-1, times=times),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+122 -121
View File
@@ -9,6 +9,7 @@ from posix_parity import cleanup_dir
from posix_parity import compare_calls
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
@@ -31,135 +32,135 @@ def has_attr(name):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_xattr <mountpoint>", file=sys.stderr)
return 1
mount = sys.argv[1]
xname = "user.posix_parity"
xvalue = b"parity-check"
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
touch(merge_file, b"x")
touch(native_file, b"x")
touch(merge_notdir, b"x")
touch(native_notdir, b"x")
err = compare_calls(
"setxattr success",
lambda: os.setxattr(merge_file, xname, xvalue),
lambda: os.setxattr(native_file, xname, xvalue),
)
if err:
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
os.setxattr(merge_file, xname, xvalue)
except OSError as exc:
err += " | " + format_oserror("mergerfs setxattr", exc)
try:
os.setxattr(native_file, xname, xvalue)
except OSError as exc:
err += " | " + format_oserror("native setxattr", exc)
return fail(err)
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
err = compare_calls(
"getxattr success",
lambda: os.getxattr(merge_file, xname),
lambda: os.getxattr(native_file, xname),
lambda lhs, rhs: lhs == rhs,
)
if err:
try:
os.getxattr(merge_file, xname)
except OSError as exc:
err += " | " + format_oserror("mergerfs getxattr", exc)
try:
os.getxattr(native_file, xname)
except OSError as exc:
err += " | " + format_oserror("native getxattr", exc)
return fail(err)
merge_file = join(merge_base, "file")
native_file = join(native_base, "file")
merge_missing = join(merge_base, "missing")
native_missing = join(native_base, "missing")
merge_notdir = join(merge_base, "notdir")
native_notdir = join(native_base, "notdir")
err = compare_calls(
"listxattr includes key",
lambda: os.listxattr(merge_file),
lambda: os.listxattr(native_file),
has_attr(xname),
)
if err:
try:
m_list = os.listxattr(merge_file)
err += f" | mergerfs list={m_list!r}"
except OSError as exc:
err += " | " + format_oserror("mergerfs listxattr", exc)
try:
n_list = os.listxattr(native_file)
err += f" | native list={n_list!r}"
except OSError as exc:
err += " | " + format_oserror("native listxattr", exc)
return fail(err)
touch(merge_file, b"x")
touch(native_file, b"x")
touch(merge_notdir, b"x")
touch(native_notdir, b"x")
err = compare_calls(
"removexattr success",
lambda: os.removexattr(merge_file, xname),
lambda: os.removexattr(native_file, xname),
)
if err:
try:
os.removexattr(merge_file, xname)
except OSError as exc:
err += " | " + format_oserror("mergerfs removexattr", exc)
try:
os.removexattr(native_file, xname)
except OSError as exc:
err += " | " + format_oserror("native removexattr", exc)
return fail(err)
xname = "user.posix_parity"
xvalue = b"parity-check"
err = compare_calls(
"getxattr missing attr",
lambda: os.getxattr(merge_file, xname),
lambda: os.getxattr(native_file, xname),
)
if err:
try:
os.getxattr(merge_file, xname)
except OSError as exc:
err += " | " + format_oserror("mergerfs getxattr missing", exc)
try:
os.getxattr(native_file, xname)
except OSError as exc:
err += " | " + format_oserror("native getxattr missing", exc)
return fail(err)
err = compare_calls(
"setxattr success",
lambda: os.setxattr(merge_file, xname, xvalue),
lambda: os.setxattr(native_file, xname, xvalue),
)
if err:
try:
os.setxattr(merge_file, xname, xvalue)
except OSError as exc:
err += " | " + format_oserror("mergerfs setxattr", exc)
try:
os.setxattr(native_file, xname, xvalue)
except OSError as exc:
err += " | " + format_oserror("native setxattr", exc)
return fail(err)
err = compare_calls(
"setxattr ENOENT",
lambda: os.setxattr(merge_missing, xname, xvalue),
lambda: os.setxattr(native_missing, xname, xvalue),
)
if err:
return fail(err)
err = compare_calls(
"getxattr success",
lambda: os.getxattr(merge_file, xname),
lambda: os.getxattr(native_file, xname),
lambda lhs, rhs: lhs == rhs,
)
if err:
try:
os.getxattr(merge_file, xname)
except OSError as exc:
err += " | " + format_oserror("mergerfs getxattr", exc)
try:
os.getxattr(native_file, xname)
except OSError as exc:
err += " | " + format_oserror("native getxattr", exc)
return fail(err)
err = compare_calls(
"getxattr ENOTDIR",
lambda: os.getxattr(join(merge_notdir, "child"), xname),
lambda: os.getxattr(join(native_notdir, "child"), xname),
)
if err:
return fail(err)
err = compare_calls(
"listxattr includes key",
lambda: os.listxattr(merge_file),
lambda: os.listxattr(native_file),
has_attr(xname),
)
if err:
try:
m_list = os.listxattr(merge_file)
err += f" | mergerfs list={m_list!r}"
except OSError as exc:
err += " | " + format_oserror("mergerfs listxattr", exc)
try:
n_list = os.listxattr(native_file)
err += f" | native list={n_list!r}"
except OSError as exc:
err += " | " + format_oserror("native listxattr", exc)
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
err = compare_calls(
"removexattr success",
lambda: os.removexattr(merge_file, xname),
lambda: os.removexattr(native_file, xname),
)
if err:
try:
os.removexattr(merge_file, xname)
except OSError as exc:
err += " | " + format_oserror("mergerfs removexattr", exc)
try:
os.removexattr(native_file, xname)
except OSError as exc:
err += " | " + format_oserror("native removexattr", exc)
return fail(err)
err = compare_calls(
"getxattr missing attr",
lambda: os.getxattr(merge_file, xname),
lambda: os.getxattr(native_file, xname),
)
if err:
try:
os.getxattr(merge_file, xname)
except OSError as exc:
err += " | " + format_oserror("mergerfs getxattr missing", exc)
try:
os.getxattr(native_file, xname)
except OSError as exc:
err += " | " + format_oserror("native getxattr missing", exc)
return fail(err)
err = compare_calls(
"setxattr ENOENT",
lambda: os.setxattr(merge_missing, xname, xvalue),
lambda: os.setxattr(native_missing, xname, xvalue),
)
if err:
return fail(err)
err = compare_calls(
"getxattr ENOTDIR",
lambda: os.getxattr(join(merge_notdir, "child"), xname),
lambda: os.getxattr(join(native_notdir, "child"), xname),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+121 -121
View File
@@ -8,6 +8,7 @@ import tempfile
from posix_parity import cleanup_dir
from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
@@ -88,155 +89,154 @@ def lremovexattr(path, name):
def main():
if len(sys.argv) != 2:
print("usage: TEST_posix_xattr_matrix <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
mount = sys.argv[1]
root_m = merge_base
root_n = native_base
with tempfile.TemporaryDirectory() as native:
merge_base = temp_dir(mount)
try:
native_base = join(native, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
file_m = join(root_m, "file")
file_n = join(root_n, "file")
dir_m = join(root_m, "dir")
dir_n = join(root_n, "dir")
link_m = join(root_m, "link")
link_n = join(root_n, "link")
root_m = merge_base
root_n = native_base
touch(file_m, b"x")
touch(file_n, b"x")
os.makedirs(dir_m, exist_ok=True)
os.makedirs(dir_n, exist_ok=True)
try:
os.unlink(link_m)
except FileNotFoundError:
pass
try:
os.unlink(link_n)
except FileNotFoundError:
pass
os.symlink("file", link_m)
os.symlink("file", link_n)
file_m = join(root_m, "file")
file_n = join(root_n, "file")
dir_m = join(root_m, "dir")
dir_n = join(root_n, "dir")
link_m = join(root_m, "link")
link_n = join(root_n, "link")
objs = [
("file", file_m, file_n),
("dir", dir_m, dir_n),
]
touch(file_m, b"x")
touch(file_n, b"x")
os.makedirs(dir_m, exist_ok=True)
os.makedirs(dir_n, exist_ok=True)
try:
os.unlink(link_m)
except FileNotFoundError:
pass
try:
os.unlink(link_n)
except FileNotFoundError:
pass
os.symlink("file", link_m)
os.symlink("file", link_n)
xbase = "user.xattr_matrix"
objs = [
("file", file_m, file_n),
("dir", dir_m, dir_n),
]
for obj_name, path_m, path_n in objs:
xname = f"{xbase}.{obj_name}"
xbase = "user.xattr_matrix"
err = compare(
f"setxattr default {obj_name}",
lambda p=path_m, n=xname: os.setxattr(p, n, b"v1"),
lambda p=path_n, n=xname: os.setxattr(p, n, b"v1"),
)
if err:
return fail(err)
for obj_name, path_m, path_n in objs:
xname = f"{xbase}.{obj_name}"
err = getxattr_cmp(path_m, path_n, xname)
if err:
return fail(err)
err = compare(
f"setxattr default {obj_name}",
lambda p=path_m, n=xname: os.setxattr(p, n, b"v1"),
lambda p=path_n, n=xname: os.setxattr(p, n, b"v1"),
)
if err:
return fail(err)
err = list_has_cmp(path_m, path_n, xname)
if err:
return fail(err)
err = getxattr_cmp(path_m, path_n, xname)
if err:
return fail(err)
if hasattr(os, "XATTR_CREATE"):
err = compare(
f"setxattr CREATE existing {obj_name}",
lambda p=path_m, n=xname: os.setxattr(p, n, b"v2", os.XATTR_CREATE),
lambda p=path_n, n=xname: os.setxattr(p, n, b"v2", os.XATTR_CREATE),
)
if err:
return fail(err)
err = list_has_cmp(path_m, path_n, xname)
if err:
return fail(err)
if hasattr(os, "XATTR_REPLACE"):
err = compare(
f"setxattr REPLACE existing {obj_name}",
lambda p=path_m, n=xname: os.setxattr(p, n, b"v3", os.XATTR_REPLACE),
lambda p=path_n, n=xname: os.setxattr(p, n, b"v3", os.XATTR_REPLACE),
)
if err:
return fail(err)
if hasattr(os, "XATTR_CREATE"):
err = getxattr_cmp(path_m, path_n, xname)
if err:
return fail(err)
missing_name = f"{xname}.missing"
if hasattr(os, "XATTR_REPLACE"):
err = compare(
f"setxattr REPLACE missing {obj_name}",
lambda p=path_m, n=missing_name: os.setxattr(p, n, b"v", os.XATTR_REPLACE),
lambda p=path_n, n=missing_name: os.setxattr(p, n, b"v", os.XATTR_REPLACE),
)
if err:
return fail(err)
err = compare(
f"removexattr existing {obj_name}",
lambda p=path_m, n=xname: os.removexattr(p, n),
lambda p=path_n, n=xname: os.removexattr(p, n),
)
if err:
return fail(err)
err = compare(
f"removexattr missing {obj_name}",
lambda p=path_m, n=xname: os.removexattr(p, n),
lambda p=path_n, n=xname: os.removexattr(p, n),
)
if err:
return fail(err)
lxname = f"{xbase}.link"
err = compare(
f"setxattr CREATE existing {obj_name}",
lambda p=path_m, n=xname: os.setxattr(p, n, b"v2", os.XATTR_CREATE),
lambda p=path_n, n=xname: os.setxattr(p, n, b"v2", os.XATTR_CREATE),
"lsetxattr symlink default",
lambda: lsetxattr(link_m, lxname, b"lv1"),
lambda: lsetxattr(link_n, lxname, b"lv1"),
)
if err:
return fail(err)
if hasattr(os, "XATTR_REPLACE"):
err = compare(
f"setxattr REPLACE existing {obj_name}",
lambda p=path_m, n=xname: os.setxattr(p, n, b"v3", os.XATTR_REPLACE),
lambda p=path_n, n=xname: os.setxattr(p, n, b"v3", os.XATTR_REPLACE),
"lgetxattr symlink",
lambda: lgetxattr(link_m, lxname),
lambda: lgetxattr(link_n, lxname),
lambda a, b: a == b,
)
if err:
return fail(err)
err = getxattr_cmp(path_m, path_n, xname)
if err:
return fail(err)
missing_name = f"{xname}.missing"
if hasattr(os, "XATTR_REPLACE"):
err = compare(
f"setxattr REPLACE missing {obj_name}",
lambda p=path_m, n=missing_name: os.setxattr(p, n, b"v", os.XATTR_REPLACE),
lambda p=path_n, n=missing_name: os.setxattr(p, n, b"v", os.XATTR_REPLACE),
"llistxattr has symlink key",
lambda: llistxattr(link_m),
lambda: llistxattr(link_n),
lambda a, b: ((lxname in a) == (lxname in b)),
)
if err:
return fail(err)
err = compare(
f"removexattr existing {obj_name}",
lambda p=path_m, n=xname: os.removexattr(p, n),
lambda p=path_n, n=xname: os.removexattr(p, n),
)
if err:
return fail(err)
err = compare(
"lremovexattr symlink",
lambda: lremovexattr(link_m, lxname),
lambda: lremovexattr(link_n, lxname),
)
if err:
return fail(err)
err = compare(
f"removexattr missing {obj_name}",
lambda p=path_m, n=xname: os.removexattr(p, n),
lambda p=path_n, n=xname: os.removexattr(p, n),
)
if err:
return fail(err)
lxname = f"{xbase}.link"
err = compare(
"lsetxattr symlink default",
lambda: lsetxattr(link_m, lxname, b"lv1"),
lambda: lsetxattr(link_n, lxname, b"lv1"),
)
if err:
return fail(err)
err = compare(
"lgetxattr symlink",
lambda: lgetxattr(link_m, lxname),
lambda: lgetxattr(link_n, lxname),
lambda a, b: a == b,
)
if err:
return fail(err)
err = compare(
"llistxattr has symlink key",
lambda: llistxattr(link_m),
lambda: llistxattr(link_n),
lambda a, b: ((lxname in a) == (lxname in b)),
)
if err:
return fail(err)
err = compare(
"lremovexattr symlink",
lambda: lremovexattr(link_m, lxname),
lambda: lremovexattr(link_n, lxname),
)
if err:
return fail(err)
return 0
finally:
cleanup_dir(merge_base)
return 0
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+141 -141
View File
@@ -8,6 +8,7 @@ import tempfile
from posix_parity import cleanup_dir
from posix_parity import join
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
@@ -16,7 +17,7 @@ libc.readlink.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_size_t]
libc.readlink.restype = ctypes.c_ssize_t
def readlink_raw(path: str, bufsiz: int):
def readlink_raw(path, bufsiz):
buf = ctypes.create_string_buffer(bufsiz)
ctypes.set_errno(0)
rv = libc.readlink(path.encode(), buf, bufsiz)
@@ -25,11 +26,11 @@ def readlink_raw(path: str, bufsiz: int):
return rv, err, bytes(buf)
def compare_case(name: str,
merge_path: str,
native_path: str,
bufsiz: int,
expect_errno: int = None):
def compare_case(name,
merge_path,
native_path,
bufsiz,
expect_errno=None):
m_rv, m_errno, m_buf = readlink_raw(merge_path, bufsiz)
n_rv, n_errno, n_buf = readlink_raw(native_path, bufsiz)
@@ -46,158 +47,157 @@ def compare_case(name: str,
def main():
if len(sys.argv) != 2:
print("usage: TEST_readlink_semantics <mountpoint>", file=sys.stderr)
return 1
try:
with mergerfs_mount() as (mount, _):
target = "target-abcdefghijklmnopqrstuvwxyz"
mount = sys.argv[1]
target = "target-abcdefghijklmnopqrstuvwxyz"
with tempfile.TemporaryDirectory() as native_dir:
merge_base = temp_dir(mount)
try:
native_base = join(native_dir, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
paths = {
"merge_link": join(merge_base, "readlink-semantics-link"),
"native_link": join(native_base, "readlink-semantics-link"),
"merge_regular": join(merge_base, "readlink-semantics-regular"),
"native_regular": join(native_base, "readlink-semantics-regular"),
"merge_notdir": join(merge_base, "readlink-semantics-notdir"),
"native_notdir": join(native_base, "readlink-semantics-notdir"),
"merge_loop": join(merge_base, "readlink-semantics-loop"),
"native_loop": join(native_base, "readlink-semantics-loop"),
"merge_private_dir": join(merge_base, "readlink-semantics-private"),
"native_private_dir": join(native_base, "readlink-semantics-private"),
}
paths["merge_private_link"] = os.path.join(paths["merge_private_dir"], "link")
paths["native_private_link"] = os.path.join(paths["native_private_dir"], "link")
for p in (
paths["merge_private_link"],
paths["merge_link"],
paths["merge_regular"],
paths["merge_notdir"],
paths["merge_loop"],
):
with tempfile.TemporaryDirectory() as native_dir:
merge_base = temp_dir(mount)
try:
os.unlink(p)
except FileNotFoundError:
pass
native_base = join(native_dir, os.path.basename(merge_base))
os.makedirs(native_base, exist_ok=True)
try:
os.rmdir(paths["merge_private_dir"])
except FileNotFoundError:
pass
except OSError:
pass
paths = {
"merge_link": join(merge_base, "readlink-semantics-link"),
"native_link": join(native_base, "readlink-semantics-link"),
"merge_regular": join(merge_base, "readlink-semantics-regular"),
"native_regular": join(native_base, "readlink-semantics-regular"),
"merge_notdir": join(merge_base, "readlink-semantics-notdir"),
"native_notdir": join(native_base, "readlink-semantics-notdir"),
"merge_loop": join(merge_base, "readlink-semantics-loop"),
"native_loop": join(native_base, "readlink-semantics-loop"),
"merge_private_dir": join(merge_base, "readlink-semantics-private"),
"native_private_dir": join(native_base, "readlink-semantics-private"),
}
paths["merge_private_link"] = os.path.join(paths["merge_private_dir"], "link")
paths["native_private_link"] = os.path.join(paths["native_private_dir"], "link")
os.symlink(target, paths["merge_link"])
os.symlink(target, paths["native_link"])
for p in (
paths["merge_private_link"],
paths["merge_link"],
paths["merge_regular"],
paths["merge_notdir"],
paths["merge_loop"],
):
try:
os.unlink(p)
except FileNotFoundError:
pass
with open(paths["merge_regular"], "w", encoding="ascii"):
pass
with open(paths["native_regular"], "w", encoding="ascii"):
pass
try:
os.rmdir(paths["merge_private_dir"])
except FileNotFoundError:
pass
except OSError:
pass
with open(paths["merge_notdir"], "w", encoding="ascii"):
pass
with open(paths["native_notdir"], "w", encoding="ascii"):
pass
os.symlink(target, paths["merge_link"])
os.symlink(target, paths["native_link"])
os.symlink("readlink-semantics-loop", paths["merge_loop"])
os.symlink("readlink-semantics-loop", paths["native_loop"])
with open(paths["merge_regular"], "w", encoding="ascii"):
pass
with open(paths["native_regular"], "w", encoding="ascii"):
pass
os.makedirs(paths["merge_private_dir"], exist_ok=True)
os.makedirs(paths["native_private_dir"], exist_ok=True)
os.symlink(target, paths["merge_private_link"])
os.symlink(target, paths["native_private_link"])
with open(paths["merge_notdir"], "w", encoding="ascii"):
pass
with open(paths["native_notdir"], "w", encoding="ascii"):
pass
cases = []
os.symlink("readlink-semantics-loop", paths["merge_loop"])
os.symlink("readlink-semantics-loop", paths["native_loop"])
for bufsiz in (0, 1, 2, 5, 8, len(target), len(target) + 1):
cases.append((
f"success/truncation bufsiz={bufsiz}",
paths["merge_link"],
paths["native_link"],
bufsiz,
errno.EINVAL if bufsiz == 0 else None,
))
os.makedirs(paths["merge_private_dir"], exist_ok=True)
os.makedirs(paths["native_private_dir"], exist_ok=True)
os.symlink(target, paths["merge_private_link"])
os.symlink(target, paths["native_private_link"])
cases.extend([
(
"EINVAL non-symlink",
paths["merge_regular"],
paths["native_regular"],
128,
errno.EINVAL,
),
(
"ENOENT missing path",
join(merge_base, "readlink-semantics-missing"),
join(native_base, "readlink-semantics-missing"),
128,
errno.ENOENT,
),
(
"ENOTDIR non-directory prefix",
os.path.join(paths["merge_notdir"], "child"),
os.path.join(paths["native_notdir"], "child"),
128,
errno.ENOTDIR,
),
(
"ELOOP prefix symlink loop",
os.path.join(paths["merge_loop"], "child"),
os.path.join(paths["native_loop"], "child"),
128,
errno.ELOOP,
),
(
"ENAMETOOLONG long pathname",
join(merge_base, "a" * 8192),
join(native_base, "a" * 8192),
128,
errno.ENAMETOOLONG,
),
])
cases = []
os.chmod(paths["merge_private_dir"], 0o600)
os.chmod(paths["native_private_dir"], 0o600)
cases.append((
"EACCES unreadable path prefix",
paths["merge_private_link"],
paths["native_private_link"],
128,
errno.EACCES if os.geteuid() != 0 else None,
))
for bufsiz in (0, 1, 2, 5, 8, len(target), len(target) + 1):
cases.append((
f"success/truncation bufsiz={bufsiz}",
paths["merge_link"],
paths["native_link"],
bufsiz,
errno.EINVAL if bufsiz == 0 else None,
))
for case in cases:
err = compare_case(*case)
if err is not None:
print(err, end="")
return 1
cases.extend([
(
"EINVAL non-symlink",
paths["merge_regular"],
paths["native_regular"],
128,
errno.EINVAL,
),
(
"ENOENT missing path",
join(merge_base, "readlink-semantics-missing"),
join(native_base, "readlink-semantics-missing"),
128,
errno.ENOENT,
),
(
"ENOTDIR non-directory prefix",
os.path.join(paths["merge_notdir"], "child"),
os.path.join(paths["native_notdir"], "child"),
128,
errno.ENOTDIR,
),
(
"ELOOP prefix symlink loop",
os.path.join(paths["merge_loop"], "child"),
os.path.join(paths["native_loop"], "child"),
128,
errno.ELOOP,
),
(
"ENAMETOOLONG long pathname",
join(merge_base, "a" * 8192),
join(native_base, "a" * 8192),
128,
errno.ENAMETOOLONG,
),
])
# Explicitly guard the prior regression.
rv, err, _ = readlink_raw(paths["merge_link"], 1)
if rv != 1 or err != 0:
print(f"bufsiz=1 expected rv=1 errno=0, got rv={rv} errno={err}", end="")
return 1
os.chmod(paths["merge_private_dir"], 0o600)
os.chmod(paths["native_private_dir"], 0o600)
cases.append((
"EACCES unreadable path prefix",
paths["merge_private_link"],
paths["native_private_link"],
128,
errno.EACCES if os.geteuid() != 0 else None,
))
return 0
finally:
try:
os.chmod(paths["merge_private_dir"], 0o700)
except (FileNotFoundError, PermissionError):
pass
for case in cases:
err = compare_case(*case)
if err is not None:
print(err, end="")
return 1
try:
os.chmod(paths["native_private_dir"], 0o700)
except (FileNotFoundError, PermissionError):
pass
rv, err, _ = readlink_raw(paths["merge_link"], 1)
if rv != 1 or err != 0:
print(f"bufsiz=1 expected rv=1 errno=0, got rv={rv} errno={err}", end="")
return 1
cleanup_dir(merge_base)
return 0
finally:
try:
os.chmod(paths["merge_private_dir"], 0o700)
except (FileNotFoundError, PermissionError):
pass
try:
os.chmod(paths["native_private_dir"], 0o700)
except (FileNotFoundError, PermissionError):
pass
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
+29 -29
View File
@@ -10,6 +10,7 @@ from posix_parity import fail
from posix_parity import join
from posix_parity import mergerfs_branches
from posix_parity import mergerfs_get_option
from posix_parity import mergerfs_mount
from posix_parity import temp_dir
from posix_parity import touch
@@ -23,41 +24,40 @@ def get_errno(func):
def main():
if len(sys.argv) != 2:
print("usage: TEST_rmdir_enotempty_priority <mountpoint>", file=sys.stderr)
return 1
mount = sys.argv[1]
merge_base = temp_dir(mount)
try:
branches = mergerfs_branches(mount)
if len(branches) < 2:
return 0
with mergerfs_mount(num_branches=2) as (mount, branches):
merge_base = temp_dir(mount)
merge_dir = join(merge_base, "nonempty_dir")
os.makedirs(merge_dir)
try:
if len(branches) < 2:
return 0
child_file = join(merge_dir, "child.txt")
branch0 = branches[0]
native_child = join(branch0, os.path.relpath(child_file, mount))
touch(child_file, b"prevents rmdir")
merge_dir = join(merge_base, "nonempty_dir")
os.makedirs(merge_dir)
err = get_errno(lambda: os.rmdir(merge_dir))
if err != errno.ENOTEMPTY:
return fail(f"rmdir non-empty dir: expected ENOTEMPTY({errno.ENOTEMPTY}), got {err}")
child_file = join(merge_dir, "child.txt")
branch0 = branches[0]
native_child = join(branch0, os.path.relpath(child_file, mount))
touch(child_file, b"prevents rmdir")
os.unlink(child_file)
err = get_errno(lambda: os.rmdir(merge_dir))
if err != 0:
return fail(f"rmdir empty dir after removing child: expected success, got errno={err}")
err = get_errno(lambda: os.rmdir(merge_dir))
if err != errno.ENOTEMPTY:
return fail(f"rmdir non-empty dir: expected ENOTEMPTY({errno.ENOTEMPTY}), got {err}")
return 0
except Exception as exc:
return fail(str(exc))
finally:
cleanup_dir(merge_base)
os.unlink(child_file)
err = get_errno(lambda: os.rmdir(merge_dir))
if err != 0:
return fail(f"rmdir empty dir after removing child: expected success, got errno={err}")
return 0
except Exception as exc:
return fail(str(exc))
finally:
cleanup_dir(merge_base)
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+32 -25
View File
@@ -1,41 +1,48 @@
#!/usr/bin/env python3
import tempfile
import os
import sys
import tempfile
from posix_parity import mergerfs_mount
def main():
(srcfd, src) = tempfile.mkstemp(dir=sys.argv[1])
(tgtfd, tgt) = tempfile.mkstemp(dir=sys.argv[1])
try:
with mergerfs_mount() as (mount, _):
(srcfd, src) = tempfile.mkstemp(dir=mount)
(tgtfd, tgt) = tempfile.mkstemp(dir=mount)
st_src_before = os.fstat(srcfd)
st_tgt_before = os.fstat(tgtfd)
st_src_before = os.fstat(srcfd)
st_tgt_before = os.fstat(tgtfd)
os.unlink(tgt)
os.rename(src, tgt)
os.unlink(tgt)
os.rename(src, tgt)
st_src_after = os.fstat(srcfd)
st_tgt_after = os.fstat(tgtfd)
st_src_after = os.fstat(srcfd)
st_tgt_after = os.fstat(tgtfd)
if st_src_after.st_size != st_src_before.st_size:
os.close(srcfd)
os.close(tgtfd)
print("rename: src fd size changed before={} after={}".format(
st_src_before.st_size, st_src_after.st_size), end='')
return 1
if st_src_after.st_size != st_src_before.st_size:
os.close(srcfd)
os.close(tgtfd)
print("rename: src fd size changed before={} after={}".format(
st_src_before.st_size, st_src_after.st_size), end='')
return 1
if st_tgt_after.st_ino != st_tgt_before.st_ino:
os.close(srcfd)
os.close(tgtfd)
print("rename: tgt fd inode changed after unlink+rename expected={} got={}".format(
st_tgt_before.st_ino, st_tgt_after.st_ino), end='')
return 1
if st_tgt_after.st_ino != st_tgt_before.st_ino:
os.close(srcfd)
os.close(tgtfd)
print("rename: tgt fd inode changed after unlink+rename expected={} got={}".format(
st_tgt_before.st_ino, st_tgt_after.st_ino), end='')
return 1
os.close(srcfd)
os.close(tgtfd)
return 0
os.close(srcfd)
os.close(tgtfd)
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+26 -19
View File
@@ -1,35 +1,42 @@
#!/usr/bin/env python3
import tempfile
import os
import sys
import tempfile
from posix_parity import mergerfs_mount
def main():
(fd, filepath) = tempfile.mkstemp(dir=sys.argv[1])
try:
with mergerfs_mount() as (mount, _):
(fd, filepath) = tempfile.mkstemp(dir=mount)
os.posix_fallocate(fd, 0, 1024)
os.posix_fallocate(fd, 0, 1024)
st1 = os.fstat(fd)
if st1.st_size < 1024:
os.close(fd)
os.unlink(filepath)
print("fallocate before unlink: expected size >= 1024, got {}".format(st1.st_size), end='')
return 1
st1 = os.fstat(fd)
if st1.st_size < 1024:
os.close(fd)
os.unlink(filepath)
print("fallocate before unlink: expected size >= 1024, got {}".format(st1.st_size), end='')
return 1
os.unlink(filepath)
os.unlink(filepath)
os.posix_fallocate(fd, 1024, 2048)
os.posix_fallocate(fd, 1024, 2048)
st2 = os.fstat(fd)
if st2.st_size < 1024 + 2048:
os.close(fd)
print("fallocate after unlink: expected size >= 3072, got {}".format(st2.st_size), end='')
return 1
st2 = os.fstat(fd)
if st2.st_size < 1024 + 2048:
os.close(fd)
print("fallocate after unlink: expected size >= 3072, got {}".format(st2.st_size), end='')
return 1
os.close(fd)
return 0
os.close(fd)
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+26 -19
View File
@@ -1,36 +1,43 @@
#!/usr/bin/env python3
import tempfile
import os
import stat
import sys
import tempfile
from posix_parity import mergerfs_mount
def main():
(fd, filepath) = tempfile.mkstemp(dir=sys.argv[1])
try:
with mergerfs_mount() as (mount, _):
(fd, filepath) = tempfile.mkstemp(dir=mount)
os.fchmod(fd, 0o700)
os.fchmod(fd, 0o700)
st1 = os.fstat(fd)
if stat.S_IMODE(st1.st_mode) != 0o700:
os.close(fd)
os.unlink(filepath)
print("fchmod before unlink: expected mode 0o700, got 0o{:o}".format(stat.S_IMODE(st1.st_mode)), end='')
return 1
st1 = os.fstat(fd)
if stat.S_IMODE(st1.st_mode) != 0o700:
os.close(fd)
os.unlink(filepath)
print("fchmod before unlink: expected mode 0o700, got 0o{:o}".format(stat.S_IMODE(st1.st_mode)), end='')
return 1
os.unlink(filepath)
os.unlink(filepath)
os.fchmod(fd, 0o755)
os.fchmod(fd, 0o755)
st2 = os.fstat(fd)
if stat.S_IMODE(st2.st_mode) != 0o755:
os.close(fd)
print("fchmod after unlink: expected mode 0o755, got 0o{:o}".format(stat.S_IMODE(st2.st_mode)), end='')
return 1
st2 = os.fstat(fd)
if stat.S_IMODE(st2.st_mode) != 0o755:
os.close(fd)
print("fchmod after unlink: expected mode 0o755, got 0o{:o}".format(stat.S_IMODE(st2.st_mode)), end='')
return 1
os.close(fd)
return 0
os.close(fd)
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+31 -21
View File
@@ -1,37 +1,47 @@
#!/usr/bin/env python3
import tempfile
import os
import sys
import tempfile
from posix_parity import mergerfs_mount
def main():
(fd, filepath) = tempfile.mkstemp(dir=sys.argv[1])
try:
with mergerfs_mount() as (mount, _):
(fd, filepath) = tempfile.mkstemp(dir=mount)
os.fchown(fd, os.getuid(), os.getgid())
uid = os.getuid()
gid = os.getgid()
st1 = os.fstat(fd)
if st1.st_uid != os.getuid() or st1.st_gid != os.getgid():
os.close(fd)
os.unlink(filepath)
print("fchown before unlink: expected uid={} gid={}, got uid={} gid={}".format(
os.getuid(), os.getgid(), st1.st_uid, st1.st_gid), end='')
return 1
os.fchown(fd, uid, gid)
os.unlink(filepath)
st1 = os.fstat(fd)
if st1.st_uid != uid or st1.st_gid != gid:
os.close(fd)
os.unlink(filepath)
print("fchown before unlink: expected uid={} gid={}, got uid={} gid={}".format(
uid, gid, st1.st_uid, st1.st_gid), end='')
return 1
os.fchown(fd, os.getuid(), os.getgid())
os.unlink(filepath)
st2 = os.fstat(fd)
if st2.st_uid != os.getuid() or st2.st_gid != os.getgid():
os.close(fd)
print("fchown after unlink: expected uid={} gid={}, got uid={} gid={}".format(
os.getuid(), os.getgid(), st2.st_uid, st2.st_gid), end='')
return 1
os.fchown(fd, uid, gid)
os.close(fd)
return 0
st2 = os.fstat(fd)
if st2.st_uid != uid or st2.st_gid != gid:
os.close(fd)
print("fchown after unlink: expected uid={} gid={}, got uid={} gid={}".format(
uid, gid, st2.st_uid, st2.st_gid), end='')
return 1
os.close(fd)
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+34 -27
View File
@@ -1,45 +1,52 @@
#!/usr/bin/env python3
import tempfile
import os
import sys
import tempfile
from posix_parity import mergerfs_mount
def main():
(fd, filepath) = tempfile.mkstemp(dir=sys.argv[1])
try:
with mergerfs_mount() as (mount, _):
(fd, filepath) = tempfile.mkstemp(dir=mount)
x_before = os.lstat(filepath)
x = os.fstat(fd)
x_before = os.lstat(filepath)
x = os.fstat(fd)
if x.st_size != x_before.st_size:
os.close(fd)
os.unlink(filepath)
print("fstat before unlink: size mismatch lstat={} fstat={}".format(x_before.st_size, x.st_size), end='')
return 1
if x.st_size != x_before.st_size:
os.close(fd)
os.unlink(filepath)
print("fstat before unlink: size mismatch lstat={} fstat={}".format(x_before.st_size, x.st_size), end='')
return 1
if x.st_mode != x_before.st_mode:
os.close(fd)
os.unlink(filepath)
print("fstat before unlink: mode mismatch lstat={} fstat={}".format(x_before.st_mode, x.st_mode), end='')
return 1
if x.st_mode != x_before.st_mode:
os.close(fd)
os.unlink(filepath)
print("fstat before unlink: mode mismatch lstat={} fstat={}".format(x_before.st_mode, x.st_mode), end='')
return 1
os.unlink(filepath)
os.unlink(filepath)
y = os.fstat(fd)
y = os.fstat(fd)
if y.st_nlink != 0:
os.close(fd)
print("fstat after unlink: expected nlink=0, got {}".format(y.st_nlink), end='')
return 1
if y.st_nlink != 0:
os.close(fd)
print("fstat after unlink: expected nlink=0, got {}".format(y.st_nlink), end='')
return 1
if y.st_size != x.st_size:
os.close(fd)
print("fstat after unlink: size changed from {} to {}".format(x.st_size, y.st_size), end='')
return 1
if y.st_size != x.st_size:
os.close(fd)
print("fstat after unlink: size changed from {} to {}".format(x.st_size, y.st_size), end='')
return 1
os.close(fd)
return 0
os.close(fd)
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+26 -19
View File
@@ -1,35 +1,42 @@
#!/usr/bin/env python3
import tempfile
import os
import sys
import tempfile
from posix_parity import mergerfs_mount
def main():
(fd, filepath) = tempfile.mkstemp(dir=sys.argv[1])
try:
with mergerfs_mount() as (mount, _):
(fd, filepath) = tempfile.mkstemp(dir=mount)
os.ftruncate(fd, 1024)
os.ftruncate(fd, 10)
st1 = os.fstat(fd)
if st1.st_size != 1024:
os.close(fd)
os.unlink(filepath)
print("ftruncate before unlink: expected size 1024, got {}".format(st1.st_size), end='')
return 1
st1 = os.fstat(fd)
if st1.st_size != 10:
os.close(fd)
os.unlink(filepath)
print("ftruncate before unlink: expected size 10, got {}".format(st1.st_size), end='')
return 1
os.unlink(filepath)
os.unlink(filepath)
os.ftruncate(fd, 2048)
os.ftruncate(fd, 20)
st2 = os.fstat(fd)
if st2.st_size != 2048:
os.close(fd)
print("ftruncate after unlink: expected size 2048, got {}".format(st2.st_size), end='')
return 1
st2 = os.fstat(fd)
if st2.st_size != 20:
os.close(fd)
print("ftruncate after unlink: expected size 20, got {}".format(st2.st_size), end='')
return 1
os.close(fd)
return 0
os.close(fd)
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+30 -25
View File
@@ -1,41 +1,46 @@
#!/usr/bin/env python3
import tempfile
import os
import sys
import tempfile
from posix_parity import mergerfs_mount
def main():
atime = 1234
mtime = 5678
(fd, filepath) = tempfile.mkstemp(dir=sys.argv[1])
try:
with mergerfs_mount() as (mount, _):
(fd, filepath) = tempfile.mkstemp(dir=mount)
os.utime(fd, (atime, mtime))
times = (1234567890, 9876543210)
os.utime(fd, times=times)
st1 = os.fstat(fd)
if st1.st_atime != atime or st1.st_mtime != mtime:
os.close(fd)
os.unlink(filepath)
print("futimens before unlink: expected atime={} mtime={}, got atime={} mtime={}".format(
atime, mtime, st1.st_atime, st1.st_mtime), end='')
return 1
st1 = os.fstat(fd)
if st1.st_atime != times[0] or st1.st_mtime != times[1]:
os.close(fd)
os.unlink(filepath)
print("futimens before unlink: expected atime={} mtime={}, got atime={} mtime={}".format(
times[0], times[1], st1.st_atime, st1.st_mtime), end='')
return 1
os.unlink(filepath)
os.unlink(filepath)
atime2 = 9999
mtime2 = 8888
os.utime(fd, (atime2, mtime2))
times2 = (1111111111, 2222222222)
os.utime(fd, times=times2)
st2 = os.fstat(fd)
if st2.st_atime != atime2 or st2.st_mtime != mtime2:
os.close(fd)
print("futimens after unlink: expected atime={} mtime={}, got atime={} mtime={}".format(
atime2, mtime2, st2.st_atime, st2.st_mtime), end='')
return 1
st2 = os.fstat(fd)
if st2.st_atime != times2[0] or st2.st_mtime != times2[1]:
os.close(fd)
print("futimens after unlink: expected atime={} mtime={}, got atime={} mtime={}".format(
times2[0], times2[1], st2.st_atime, st2.st_mtime), end='')
return 1
os.close(fd)
return 0
os.close(fd)
return 0
except RuntimeError as exc:
print(str(exc), end="")
return 77
if __name__ == "__main__":
raise SystemExit(main())
raise SystemExit(main())
+73
View File
@@ -5,7 +5,11 @@ import errno
import os
import shutil
import stat
import subprocess
import sys
import tempfile
import time
from contextlib import contextmanager
libc = ctypes.CDLL(None, use_errno=True)
@@ -13,6 +17,75 @@ libc.access.argtypes = [ctypes.c_char_p, ctypes.c_int]
libc.access.restype = ctypes.c_int
def find_mergerfs():
script_dir = os.path.dirname(os.path.realpath(__file__))
build_path = os.path.join(script_dir, "..", "build", "mergerfs")
if os.path.isfile(build_path):
return os.path.realpath(build_path)
return shutil.which("mergerfs")
def find_fusermount():
return shutil.which("fusermount3") or shutil.which("fusermount")
def has_tool(name):
return shutil.which(name) is not None
def mount_mergerfs(branches, mountpoint, options=None):
mergerfs_bin = find_mergerfs()
if mergerfs_bin is None:
raise RuntimeError("mergerfs binary not found")
if options is None:
options = "defaults,use_ino,category.create=mfs"
if os.geteuid() == 0:
options += ",allow_other"
branch_arg = ":".join(branches)
cmd = [mergerfs_bin, branch_arg, mountpoint, "-o", options]
subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
def unmount_mergerfs(mountpoint):
um = find_fusermount()
if um is None:
raise RuntimeError("fusermount3/fusermount not found")
for attempt in range(3):
u = subprocess.run([um, "-u", mountpoint], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if u.returncode == 0:
return
time.sleep(1)
subprocess.run([um, "-z", mountpoint], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
@contextmanager
def mergerfs_mount(num_branches=2, options=None):
mergerfs_bin = find_mergerfs()
if mergerfs_bin is None:
raise RuntimeError("mergerfs binary not found")
if find_fusermount() is None:
raise RuntimeError("fusermount3/fusermount not found")
tests_dir = os.path.dirname(os.path.realpath(__file__))
tmp_parent = os.path.join(tests_dir, ".test_tmp")
os.makedirs(tmp_parent, exist_ok=True)
tmpdir = tempfile.mkdtemp(prefix="mergerfs_test_", dir=tmp_parent)
try:
branches = [os.path.join(tmpdir, f"branch{i}") for i in range(num_branches)]
for b in branches:
os.makedirs(b, exist_ok=True)
mountpoint = os.path.join(tmpdir, "mp")
os.makedirs(mountpoint, exist_ok=True)
mount_mergerfs(branches, mountpoint, options)
yield mountpoint, branches
finally:
try:
unmount_mergerfs(mountpoint)
except Exception:
pass
shutil.rmtree(tmpdir, ignore_errors=True)
def temp_dir(root):
"""Create a temporary directory within the given root directory."""
return tempfile.mkdtemp(dir=root)
+2 -6
View File
@@ -8,10 +8,6 @@ import subprocess
SKIP = 77
if len(sys.argv) != 2:
print('usage: run-tests <mountpoint>\n', file=sys.stderr)
sys.exit(1)
test_path = os.path.realpath(sys.argv[0])
test_path = os.path.dirname(test_path)
@@ -22,9 +18,9 @@ for entry in os.scandir(test_path):
continue
try:
print(entry.name + ': ', end='')
print(entry.name + ': ', end='', flush=True)
fullpath = os.path.join(test_path, entry.name)
args = [fullpath, sys.argv[1]]
args = [fullpath]
rv = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output, _ = rv.communicate(timeout=120)
if rv.returncode == SKIP:
+56
View File
@@ -2730,6 +2730,61 @@ test_config_set_kv_form()
TEST_CHECK(val == "true");
}
void
test_config_remember_nodes_bool_numeric()
{
bool old_remember_nodes = fuse_cfg.remember_nodes;
Config cfg;
std::string val;
fuse_cfg.remember_nodes = false;
// remember-nodes accepts ints: 0 disables, non-zero enables
TEST_CHECK(cfg.set("remember-nodes", "0") == 0);
TEST_CHECK(cfg.get("remember-nodes", &val) == 0);
TEST_CHECK(val == "false");
TEST_CHECK(cfg.set("remember-nodes", "86400") == 0);
TEST_CHECK(cfg.get("remember-nodes", &val) == 0);
TEST_CHECK(val == "true");
TEST_CHECK(cfg.set("remember", "-1") == 0);
TEST_CHECK(cfg.get("remember", &val) == 0);
TEST_CHECK(val == "true");
// remember-nodes also accepts bool strings
TEST_CHECK(cfg.set("remember-nodes", "false") == 0);
TEST_CHECK(cfg.get("remember-nodes", &val) == 0);
TEST_CHECK(val == "false");
// never-forget-nodes is bool; it reflects the feature state
TEST_CHECK(cfg.set("never-forget-nodes", "true") == 0);
TEST_CHECK(cfg.get("never-forget-nodes", &val) == 0);
TEST_CHECK(val == "true");
// Querying remember-nodes after never-forget-nodes=true returns true
// (both alias the same fuse_cfg.remember_nodes)
TEST_CHECK(cfg.get("remember-nodes", &val) == 0);
TEST_CHECK(val == "true");
// Empty string treated as true
TEST_CHECK(cfg.set("noforget", "") == 0);
TEST_CHECK(cfg.get("noforget", &val) == 0);
TEST_CHECK(val == "true");
// remember-nodes also accepts empty string as true
fuse_cfg.remember_nodes = false;
TEST_CHECK(cfg.set("remember-nodes", "") == 0);
TEST_CHECK(cfg.get("remember-nodes", &val) == 0);
TEST_CHECK(val == "true");
// Invalid string rejected by remember-nodes
TEST_CHECK(cfg.set("remember-nodes", "notabool") == -EINVAL);
fuse_cfg.remember_nodes = old_remember_nodes;
}
void
test_config_readonly_before_init()
{
@@ -3683,6 +3738,7 @@ TEST_LIST =
{"config_set_underscore_normalization",test_config_set_underscore_normalization},
{"config_get_underscore_normalization",test_config_get_underscore_normalization},
{"config_set_kv_form",test_config_set_kv_form},
{"config_remember_nodes_bool_numeric",test_config_remember_nodes_bool_numeric},
{"config_readonly_before_init",test_config_readonly_before_init},
{"config_readonly_after_init",test_config_readonly_after_init},
{"config_mutable_after_init",test_config_mutable_after_init},
+23 -15
View File
@@ -32,9 +32,6 @@ UTILS :=
INSTALLUTILS :=
endif
ifeq ($(NDEBUG),1)
OPT_FLAGS := -O2 -DNDEBUG
else
ifdef SANITIZE
ifeq ($(SANITIZE),1)
override SANITIZE := -fsanitize=address,undefined,leak
@@ -42,15 +39,22 @@ else ifeq ($(filter -fsanitize=%,$(SANITIZE)),)
override SANITIZE := -fsanitize=$(SANITIZE)
endif
endif
OPT_FLAGS := -O0 \
-g \
-fno-omit-frame-pointer \
$(SANITIZE) \
-fstack-protector-strong \
-Wextra \
-Werror \
-Wno-unused-parameter \
-DDEBUG
COMMON_OPT_FLAGS += \
-fstack-protector-strong \
-Wextra \
-Wno-unused-parameter
DEBUG_OPT_FLAGS += \
-O0 \
-g3 \
-fno-omit-frame-pointer \
$(SANITIZE)
ifdef RELEASE
OPT_FLAGS := -O3 $(COMMON_OPT_FLAGS)
else
OPT_FLAGS ?= $(DEBUG_OPT_FLAGS) $(COMMON_OPT_FLAGS)
endif
ifeq ($(LTO),1)
@@ -110,7 +114,7 @@ CXXFLAGS ?= \
-Wall \
-pipe
override CXXFLAGS += \
-std=c++17 \
-std=c++20 \
-MMD \
-MP
FUSERMOUNT_DIR := $(BINDIR)
@@ -151,10 +155,10 @@ $(BUILDDIR)/mount.mergerfs: $(BUILDDIR)/libfuse.a util/mount.mergerfs.cpp
mount.mergerfs: $(BUILDDIR)/mount.mergerfs
$(BUILDDIR)/.objs/%.c.o: lib/%.c | $(BUILDDIR)/stamp
$(CC) $(CFLAGS) $(FUSE_FLAGS) -c $< -o $@
$(CC) $(CFLAGS) $(CPPFLAGS) $(FUSE_FLAGS) -c $< -o $@
$(BUILDDIR)/.objs/%.cpp.o: lib/%.cpp | $(BUILDDIR)/stamp
$(CXX) $(CXXFLAGS) $(FUSE_FLAGS) -c $< -o $@
$(CXX) $(CXXFLAGS) $(CPPFLAGS) $(FUSE_FLAGS) -c $< -o $@
.PHONY: clean
clean:
@@ -176,6 +180,10 @@ install-utils: mergerfs-fusermount mount.mergerfs strip
install: $(INSTALLUTILS)
.PHONY: release
release:
$(MAKE) RELEASE=1 NDEBUG=1 CPPFLAGS="$(CPPFLAGS) -DNDEBUG" all
.PHONY: objects strip utils install install-utils
-include $(DEPS)
+5 -4
View File
@@ -27,7 +27,8 @@ Because the high-level interface is path based, it is not possible to
delegate looking up by inode to the filesystem.
To work around this, currently a "noforget" option is provided, which
makes the library remember nodes forever. This will make the NFS
server happy, but also results in an ever growing memory footprint for
the filesystem. For this reason if the filesystem is large (or the
memory is small), then this option is not recommended.
makes the library remember nodes until they are unlinked or otherwise
removed. This will make the NFS server happy, but also results in a
larger memory footprint for the filesystem. For this reason if the
filesystem is large (or the memory is small), then this option is not
recommended.
+2 -11
View File
@@ -14,7 +14,6 @@
#include "fuse_kernel.h"
#include "fuse_req_ctx.h"
#include <fcntl.h>
#include <time.h>
#include <utime.h>
#include <sys/types.h>
@@ -176,10 +175,6 @@ struct fuse_operations
const uint64_t,
struct stat *,
fuse_timeouts_t *);
int (*lock)(const fuse_req_ctx_t *,
const fuse_file_info_t *,
int cmd,
struct flock *);
int (*utimens)(const fuse_req_ctx_t *,
const char *,
const struct timespec tv[2]);
@@ -211,9 +206,6 @@ struct fuse_operations
char *buf,
size_t size,
off_t off);
int (*flock)(const fuse_req_ctx_t *,
const fuse_file_info_t *,
int op);
int (*fallocate)(const fuse_req_ctx_t *,
const uint64_t,
int,
@@ -373,13 +365,12 @@ void fuse_populate_maintenance_thread(struct fuse *fuse);
/**
* Iterate over cache removing stale entries
* use in conjunction with "-oremember"
* Legacy cache cleanup entry point.
*
* NOTE: This is already done for the standard sessions
*
* @param fuse struct fuse pointer for fuse instance
* @return the number of seconds until the next cleanup
* @return 0
*/
int fuse_clean_cache(struct fuse *fuse);
+3 -1
View File
@@ -5,6 +5,7 @@
#include <climits>
#include <cstdio>
#include <memory>
#include <shared_mutex>
#include <string>
#define FUSE_CFG_INVALID_ID -1
@@ -25,7 +26,7 @@ struct fuse_cfg_t
s64 umask = FUSE_CFG_INVALID_UMASK;
bool valid_umask() const;
s64 remember_nodes = 0;
bool remember_nodes = false;
bool debug = false;
std::shared_ptr<FILE> log_file() const;
@@ -33,6 +34,7 @@ struct fuse_cfg_t
std::shared_ptr<std::string> log_filepath() const;
void log_filepath(const std::string &);
private:
mutable std::shared_mutex _log_mutex;
std::shared_ptr<FILE> _log_file = sp<FILE>(stderr,[](FILE*){});
std::shared_ptr<std::string> _log_filepath = sp<std::string>();

Some files were not shown because too many files have changed in this diff Show More