fuse_open.cpp

If using Linux 6.9 or above and enabled (passthrough=true) files
opened or created will use the FUSE passthrough feature.

If direct-io=true / cache.files=off it will override passthrough. If
direct-io-allow-mmap is enabled only mmap will passthrough.

HANDLE_KILLPRIV and V2 are enabled now by default to remove the
kernel's need to issue getattr and getxattr requests.

moveonenospc will not work when leveraging passthrough.

README.md

@@ -89,7 +89,12 @@ start with one of the following option sets.
 
 #### You need `mmap` (used by rtorrent and many sqlite3 base software)
 
-`cache.files=partial,dropcacheonclose=true,category.create=mfs`
+`cache.files=auto-full,dropcacheonclose=true,category.create=mfs`
+
+or if you are on a Linux kernel >= 6.6.x mergerfs will enable a mode
+that allows shared mmap when `cache.files=off`. To be sure of the best
+performance between `cache.files=off` and `cache.files=auto-full`
+you'll need to do your own benchmarking but often `off` is faster.
 
 #### You don't need `mmap`
 
@@ -97,11 +102,11 @@ start with one of the following option sets.
 
 ### Command Line
 
-`mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media`
+`mergerfs -o cache.files=auto-full,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media`
 
 ### /etc/fstab
 
-`/mnt/hdd0:/mnt/hdd1 /media fuse.mergerfs cache.files=partial,dropcacheonclose=true,category.create=mfs 0 0`
+`/mnt/hdd0:/mnt/hdd1 /media mergerfs cache.files=auto-full,dropcacheonclose=true,category.create=mfs 0 0`
 
 ### systemd mount
 
@@ -116,7 +121,7 @@ Type=simple
 KillMode=none
 ExecStart=/usr/bin/mergerfs \
   -f \
-  -o cache.files=partial \
+  -o cache.files=auto-full \
   -o dropcacheonclose=true \
   -o category.create=mfs \
   /mnt/hdd0:/mnt/hdd1 \
@@ -162,6 +167,14 @@ These options are the same regardless of whether you use them with the
   longer need the data and it can drop its cache. Recommended when
   **cache.files=partial|full|auto-full|per-process** to limit double
   caching. (default: false)
+* **passthrough=off|ro|wo|rw**: On Linux 6.9 and above passthrough allows for
+  near native IO performance. See below for more details. (default:
+  off)
+* **direct-io-allow-mmap=BOOL**: On Linux 6.6 and above it is
+  possible to disable file page caching while still allowing for
+  shared mmap support. mergerfs will enable this feature if available
+  but an option is provided to turn it off for testing and debugging
+  purposes. (default: true)
 * **symlinkify=BOOL**: When enabled and a file is not writable and its
   mtime or ctime is older than **symlinkify_timeout** files will be
   reported as symlinks to the original files. Please read more below
@@ -388,8 +401,8 @@ The above line will use all mount points in /mnt prefixed with **hdd** and **ssd
 To have the pool mounted at boot or otherwise accessible from related tools use **/etc/fstab**.
 
 ```
-# <file system>        <mount point>  <type>         <options>             <dump>  <pass>
-/mnt/hdd*:/mnt/ssd    /media          fuse.mergerfs  minfreespace=16G      0       0
+# <file system>        <mount point>  <type>    <options>             <dump>  <pass>
+/mnt/hdd*:/mnt/ssd    /media          mergerfs  minfreespace=16G      0       0
 ```
 
 **NOTE:** the globbing is done at mount or when updated using the
@@ -401,6 +414,32 @@ fact it will not be automatically included.
 **fuse** package.
 
 
+### passthrough
+
+With Linux 6.9 and above there is a feature in FUSE called
+"passthrough" which can improve performance by allowing a union or
+overlay filesystem (like mergerfs) to pass the file descriptor of a
+file opened on an underlying filesystem to the kernel to work on
+directly rather than calls always having to go to the FUSE server
+(mergerfs). As of Linux 6.9 the functions that can be passthroughed
+are regular file read/write IO and mmap. Other functions may be added
+in the future.
+
+Passthrough requires `cache.files` to be enabled. `cache.files=off`
+(which enables FUSE direct-io feature) will override `passthrough` and
+read/write requests will still be sent to mergerfs.
+
+If `cache.files=off` and `direct-io-allow-mmap=true` (the default)
+then regular read/write IO will be handled by mergerfs but mmap will
+be passthroughed.
+
+NOTE: `moveonenospc` will not work when using `passthrough` as it
+removes mergerfs entirely from the read/write process. If an ENOSPC
+error occurs it will be returned to the caller app immediately. Since
+mergerfs won't be handling the IO it therefore won't be able to handle
+any error returned by said IO.
+
+
 ### inodecalc
 
 Inodes (st_ino) are unique identifiers within a filesystem. Each
@@ -682,21 +721,13 @@ is empty is opened for writing.
 In theory this flag should not be exposed to the end user. It is a
 low-level FUSE flag which indicates whether or not the kernel can send
 certain kinds of messages to it for the purposes of using with
-NFS. mergerfs does support these messages but due to a possible bug in
-the Linux kernel this option needs to be configurable. As it turns out
-at least certain versions of NFS can still work when disabled and no
-longer triggers the bug.
-
-While it may be safe to default this setting to `false` in an
-abundance of caution it is being left enabled by default.
+NFS. mergerfs does support these messages but due bugs and quirks
+found in the kernel and mergerfs this option is provided just in case
+it is needed for debugging.
 
 Given that this flag is set when the FUSE connection is first
 initiated it is not possible to change during run time.
 
-See [Kernel Issues &
-Bugs](https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs)
-for more details.
-
 
 # FUNCTIONS, CATEGORIES and POLICIES
 
@@ -1178,8 +1209,8 @@ mount. If you plan on changing settings with the new mount you should
 ```
 $ sudo mount /mnt/mergerfs
 $ mount | grep mergerfs
-media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
-media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
+media on /mnt/mergerfs type mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
+media on /mnt/mergerfs type mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
 ```
 
 A problem with this approach is that the underlying instance will
@@ -1662,6 +1693,8 @@ over the suggestions below (including the benchmarking section.)
 NOTE: be sure to read about these features before changing them to
 understand what behaviors it may impact
 
+* test theoretical performance using `nullrw` or mounting a ram disk
+* enable `passthrough`
 * disable `security_capability` and/or `xattr`
 * increase cache timeouts `cache.attr`, `cache.entry`, `cache.negative_entry`
 * enable (or disable) page caching (`cache.files`)
@@ -1673,7 +1706,6 @@ understand what behaviors it may impact
 * change the number of worker threads
 * disable `posix_acl`
 * disable `async_read`
-* test theoretical performance using `nullrw` or mounting a ram disk
 * use `symlinkify` if your data is largely static and read-only
 * use tiered cache devices
 * use LVM and LVM cache to place a SSD in front of your HDDs
@@ -1876,16 +1908,17 @@ more details.
 
 #### rtorrent fails with ENODEV (No such device)
 
-Be sure to set `cache.files=partial|full|auto-full|per-processe` or
-turn off `direct_io`. rtorrent and some other applications use
-[mmap](http://linux.die.net/man/2/mmap) to read and write to files and
-offer no fallback to traditional methods. FUSE does not currently
-support mmap while using `direct_io`. There may be a performance
-penalty on writes with `direct_io` off as well as the problem of
-double caching but it's the only way to get such applications to
-work. If the performance loss is too high for other apps you can mount
-mergerfs twice. Once with `direct_io` enabled and one without it. Be
-sure to set `dropcacheonclose=true` if not using `direct_io`.
+Be sure to set
+`cache.files=partial|full|auto-full|per-process`. rtorrent and some
+other applications use [mmap](http://linux.die.net/man/2/mmap) to read
+and write to files and offer no fallback to traditional methods. FUSE
+does not currently support mmap while using `direct_io`. There may be
+a performance penalty on writes with `direct_io` off as well as the
+problem of double caching but it's the only way to get such
+applications to work. If the performance loss is too high for other
+apps you can mount mergerfs twice. Once with `direct_io` enabled and
+one without it. Be sure to set `dropcacheonclose=true` if not using
+`direct_io`.
 
 
 #### Plex doesn't work with mergerfs
@@ -2047,17 +2080,19 @@ remote filesystem on Unix/POSIX systems. Due to how NFS works there
 are some settings which need to be set in order for mergerfs to work
 with it.
 
-It should be noted that NFS and FUSE (the technology mergerfs uses)
-do not work perfectly with one another due to certain design
-choices in FUSE (and mergerfs.) Due to these issues it is
-generally recommended to use SMB when possible till situations
-change. That said issues should still be reported. NFS is not really
-recommended but it isn't unsupported.
+It should be noted that NFS and FUSE (the technology mergerfs uses) do
+not work perfectly with one another due to certain design choices in
+FUSE (and mergerfs.) Due to these issues it is generally recommended
+to use SMB when possible till situations change. That said mergerfs
+should generally work as an export of NFS and issues discovered should
+still be reported.
+
+To ensure compatibility between mergerfs and NFS use the following
+settings.
 
 mergerfs settings:
 * noforget
 * inodecalc=path-hash
-* export-support=false (only available in v2.40.0 and above)
 
 NFS export settings:
 * fsid=UUID
@@ -2080,32 +2115,17 @@ calculation algorithm then it is possible that if you changed a file
 or updated a directory the file mergerfs will use will be on a
 different branch and therefore the inode would change. This isn't an
 ideal solution and others are being considered but it works for most
-situations. It might be possible to leave the default value of
-`hybrid-hash` if `export-support=false` but it hasn't been fully
-tested yet.
-
-`export-support=false` is a low-level FUSE option that is needed as a
-workaround to a Linux kernel bug. Technically this option should be
-set to `true` when expecting to use mergerfs with NFS but it turns out
-that at least some versions of NFS work when not enabled. At the
-moment it is unclear if this has any negative side effects but when
-disabled NFS appears to work alright and the kernel no longer sends
-invalid requests. When the issue is resolved in the kernel these docs
-will be updated with appropriate details.
-
-Also see [Kernel Issues &
-Bugs](https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs)
-for more details.
+situations.
 
 `fsid=UUID` is needed because FUSE filesystems don't have different
 `st_dev` values which can cause issues when exporting. The easiest
 thing to do is set each mergerfs export `fsid` to some random
 value. An easy way to generate a random value is to use the command
-line tool `uuidgen` or through a website such as
+line tool `uuid` or `uuidgen` or through a website such as
 [uuidgenerator.net](https://www.uuidgenerator.net/).
 
 `no_root_squash` is not strictly necessary but can lead to confusing
-permission and ownership issues.
+permission and ownership issues if root squashing is enabled.
 
 
 ## SMB / CIFS

libfuse/include/fuse.h

@@ -667,6 +667,8 @@ int fuse_loop_mt(struct fuse *f);
  */
 struct fuse_context *fuse_get_context(void);
 
+int fuse_get_dev_fuse_fd(const struct fuse_context *);
+
 /**
  * Check if the current request has already been interrupted
  *
@@ -779,6 +781,11 @@ void fuse_gc1();
 void fuse_gc();
 void fuse_invalidate_all_nodes();
 
+int fuse_passthrough_open(const struct fuse_context *fc,
+                          const int                  fd);
+int fuse_passthrough_close(const struct fuse_context *fc,
+                           const int                  backing_id);
+
 EXTERN_C_END
 
 #endif /* _FUSE_H_ */

libfuse/include/fuse_common.h

@@ -88,43 +88,41 @@ struct fuse_file_info_t
 
   uint32_t noflush:1;
 
+  uint32_t passthrough:1;
+
   /** File handle.  May be filled in by filesystem in open().
       Available in all other file operations */
   uint64_t fh;
 
+  uint32_t backing_id;
+
   /** Lock owner id.  Available in locking operations and flush */
   uint64_t lock_owner;
 };
 
-/**
- * Capability bits for 'fuse_conn_info.capable' and 'fuse_conn_info.want'
- *
- * FUSE_CAP_ASYNC_READ: filesystem supports asynchronous read requests
- * FUSE_CAP_POSIX_LOCKS: filesystem supports "remote" locking
- * FUSE_CAP_ATOMIC_O_TRUNC: filesystem handles the O_TRUNC open flag
- * FUSE_CAP_EXPORT_SUPPORT: filesystem handles lookups of "." and ".."
- * FUSE_CAP_BIG_WRITES: filesystem can handle write size larger than 4kB
- * FUSE_CAP_DONT_MASK: don't apply umask to file mode on create operations
- * FUSE_CAP_IOCTL_DIR: ioctl support on directories
- * FUSE_CAP_CACHE_SYMLINKS: cache READLINK responses
- */
-#define FUSE_CAP_ASYNC_READ        (1 << 0)
-#define FUSE_CAP_POSIX_LOCKS       (1 << 1)
-#define FUSE_CAP_ATOMIC_O_TRUNC    (1 << 3)
-#define FUSE_CAP_EXPORT_SUPPORT    (1 << 4)
-#define FUSE_CAP_BIG_WRITES        (1 << 5)
-#define FUSE_CAP_DONT_MASK         (1 << 6)
-#define FUSE_CAP_FLOCK_LOCKS       (1 << 10)
-#define FUSE_CAP_IOCTL_DIR         (1 << 11)
-#define FUSE_CAP_READDIR_PLUS      (1 << 13)
-#define FUSE_CAP_READDIR_PLUS_AUTO (1 << 14)
-#define FUSE_CAP_ASYNC_DIO         (1 << 15)
-#define FUSE_CAP_WRITEBACK_CACHE   (1 << 16)
-#define FUSE_CAP_PARALLEL_DIROPS   (1 << 18)
-#define FUSE_CAP_POSIX_ACL         (1 << 19)
-#define FUSE_CAP_CACHE_SYMLINKS    (1 << 20)
-#define FUSE_CAP_MAX_PAGES         (1 << 21)
-#define FUSE_CAP_SETXATTR_EXT      (1 << 22)
+#define FUSE_CAP_ASYNC_READ           (1ULL << 0)
+#define FUSE_CAP_POSIX_LOCKS          (1ULL << 1)
+#define FUSE_CAP_ATOMIC_O_TRUNC       (1ULL << 3)
+#define FUSE_CAP_EXPORT_SUPPORT       (1ULL << 4)
+#define FUSE_CAP_BIG_WRITES           (1ULL << 5)
+#define FUSE_CAP_DONT_MASK            (1ULL << 6)
+#define FUSE_CAP_FLOCK_LOCKS          (1ULL << 10)
+#define FUSE_CAP_IOCTL_DIR            (1ULL << 11)
+#define FUSE_CAP_READDIR_PLUS         (1ULL << 13)
+#define FUSE_CAP_READDIR_PLUS_AUTO    (1ULL << 14)
+#define FUSE_CAP_ASYNC_DIO            (1ULL << 15)
+#define FUSE_CAP_WRITEBACK_CACHE      (1ULL << 16)
+#define FUSE_CAP_PARALLEL_DIROPS      (1ULL << 18)
+#define FUSE_CAP_POSIX_ACL            (1ULL << 19)
+#define FUSE_CAP_CACHE_SYMLINKS       (1ULL << 20)
+#define FUSE_CAP_MAX_PAGES            (1ULL << 21)
+#define FUSE_CAP_SETXATTR_EXT         (1ULL << 22)
+#define FUSE_CAP_DIRECT_IO_ALLOW_MMAP (1ULL << 23)
+#define FUSE_CAP_CREATE_SUPP_GROUP    (1ULL << 24)
+#define FUSE_CAP_PASSTHROUGH          (1ULL << 25)
+#define FUSE_CAP_HANDLE_KILLPRIV      (1ULL << 26)
+#define FUSE_CAP_HANDLE_KILLPRIV_V2   (1ULL << 27)
+
 
 /**
  * Ioctl flags
@@ -150,56 +148,17 @@ struct fuse_file_info_t
  * indicate the value requested by the filesystem.  The requested
  * value must usually be smaller than the indicated value.
  */
-struct fuse_conn_info {
-  /**
-   * Major version of the protocol (read-only)
-   */
+struct fuse_conn_info
+{
   unsigned proto_major;
-
-  /**
-   * Minor version of the protocol (read-only)
-   */
   unsigned proto_minor;
-
-  /**
-   * Maximum size of the write buffer
-   */
   unsigned max_write;
-
-  /**
-   * Maximum readahead
-   */
   unsigned max_readahead;
-
-  /**
-   * Capability flags, that the kernel supports
-   */
-  unsigned capable;
-
-  /**
-   * Capability flags, that the filesystem wants to enable
-   */
-  unsigned want;
-
-  /**
-   * Maximum number of backgrounded requests
-   */
+  uint64_t capable;
+  uint64_t want;
   unsigned max_background;
-
-  /**
-   * Kernel congestion threshold parameter
-   */
   unsigned congestion_threshold;
-
-  /**
-   * Max pages
-   */
   uint16_t max_pages;
-
-  /**
-   * For future use.
-   */
-  unsigned reserved[22];
 };
 
 struct fuse_session;

libfuse/include/fuse_kernel.h

@@ -211,6 +211,12 @@
  *  7.39
  *  - add FUSE_DIRECT_IO_ALLOW_MMAP
  *  - add FUSE_STATX and related structures
+ *
+ *  7.40
+ *  - add max_stack_depth to fuse_init_out, add FUSE_PASSTHROUGH init flag
+ *  - add backing_id to fuse_open_out, add FOPEN_PASSTHROUGH open flag
+ *  - add FUSE_NO_EXPORT_SUPPORT init flag
+ *  - add FUSE_NOTIFY_RESEND, add FUSE_HAS_RESEND init flag
  */
 
 #ifndef _LINUX_FUSE_H
@@ -246,7 +252,7 @@
 #define FUSE_KERNEL_VERSION 7
 
 /** Minor version number of this interface */
-#define FUSE_KERNEL_MINOR_VERSION 39
+#define FUSE_KERNEL_MINOR_VERSION 40
 
 /** The node ID of the root inode */
 #define FUSE_ROOT_ID 1
@@ -353,6 +359,7 @@ struct fuse_file_lock {
  * FOPEN_STREAM: the file is stream-like (no file position at all)
  * FOPEN_NOFLUSH: don't flush data cache on close (unless FUSE_WRITEBACK_CACHE)
  * FOPEN_PARALLEL_DIRECT_WRITES: Allow concurrent direct writes on the same inode
+ * FOPEN_PASSTHROUGH: passthrough read/write io for this open file
  */
 #define FOPEN_DIRECT_IO		(1 << 0)
 #define FOPEN_KEEP_CACHE	(1 << 1)
@@ -361,6 +368,7 @@ struct fuse_file_lock {
 #define FOPEN_STREAM		(1 << 4)
 #define FOPEN_NOFLUSH		(1 << 5)
 #define FOPEN_PARALLEL_DIRECT_WRITES	(1 << 6)
+#define FOPEN_PASSTHROUGH	(1 << 7)
 
 /**
  * INIT request/reply flags
@@ -410,6 +418,9 @@ struct fuse_file_lock {
  *			symlink and mknod (single group that matches parent)
  * FUSE_HAS_EXPIRE_ONLY: kernel supports expiry-only entry invalidation
  * FUSE_DIRECT_IO_ALLOW_MMAP: allow shared mmap in FOPEN_DIRECT_IO mode.
+ * FUSE_NO_EXPORT_SUPPORT: explicitly disable export support
+ * FUSE_HAS_RESEND: kernel supports resending pending requests, and the high bit
+ *		    of the request ID indicates resend requests
  */
 #define FUSE_ASYNC_READ		(1 << 0)
 #define FUSE_POSIX_LOCKS	(1 << 1)
@@ -449,6 +460,9 @@ struct fuse_file_lock {
 #define FUSE_CREATE_SUPP_GROUP	(1ULL << 34)
 #define FUSE_HAS_EXPIRE_ONLY	(1ULL << 35)
 #define FUSE_DIRECT_IO_ALLOW_MMAP (1ULL << 36)
+#define FUSE_PASSTHROUGH	(1ULL << 37)
+#define FUSE_NO_EXPORT_SUPPORT	(1ULL << 38)
+#define FUSE_HAS_RESEND		(1ULL << 39)
 
 /* Obsolete alias for FUSE_DIRECT_IO_ALLOW_MMAP */
 #define FUSE_DIRECT_IO_RELAX	FUSE_DIRECT_IO_ALLOW_MMAP
@@ -635,6 +649,7 @@ enum fuse_notify_code {
 	FUSE_NOTIFY_STORE = 4,
 	FUSE_NOTIFY_RETRIEVE = 5,
 	FUSE_NOTIFY_DELETE = 6,
+	FUSE_NOTIFY_RESEND = 7,
 	FUSE_NOTIFY_CODE_MAX,
 };
 
@@ -761,7 +776,7 @@ struct fuse_create_in {
 struct fuse_open_out {
 	uint64_t	fh;
 	uint32_t	open_flags;
-	uint32_t	padding;
+	int32_t		backing_id;
 };
 
 struct fuse_release_in {
@@ -877,7 +892,8 @@ struct fuse_init_out {
 	uint16_t	max_pages;
 	uint16_t	map_alignment;
 	uint32_t	flags2;
-	uint32_t	unused[7];
+	uint32_t	max_stack_depth;
+	uint32_t	unused[6];
 };
 
 #define CUSE_INIT_INFO_MAX 4096
@@ -960,6 +976,14 @@ struct fuse_fallocate_in {
 	uint32_t	padding;
 };
 
+/**
+ * FUSE request unique ID flag
+ *
+ * Indicates whether this is a resend request. The receiver should handle this
+ * request accordingly.
+ */
+#define FUSE_UNIQUE_RESEND (1ULL << 63)
+
 struct fuse_in_header {
 	uint32_t	len;
 	uint32_t	opcode;
@@ -1049,9 +1073,18 @@ struct fuse_notify_retrieve_in {
 	uint64_t	dummy4;
 };
 
+struct fuse_backing_map {
+	int32_t		fd;
+	uint32_t	flags;
+	uint64_t	padding;
+};
+
 /* Device ioctls: */
 #define FUSE_DEV_IOC_MAGIC		229
 #define FUSE_DEV_IOC_CLONE		_IOR(FUSE_DEV_IOC_MAGIC, 0, uint32_t)
+#define FUSE_DEV_IOC_BACKING_OPEN	_IOW(FUSE_DEV_IOC_MAGIC, 1, \
+					     struct fuse_backing_map)
+#define FUSE_DEV_IOC_BACKING_CLOSE	_IOW(FUSE_DEV_IOC_MAGIC, 2, uint32_t)
 
 struct fuse_lseek_in {
 	uint64_t	fh;

libfuse/lib/debug.c

@@ -158,9 +158,9 @@ open_flag_to_str(const uint64_t offset_)
 static
 const
 char*
-fuse_flag_to_str(const uint32_t offset_)
+fuse_flag_to_str(const uint64_t offset_)
 {
-  switch(1 << offset_)
+  switch(1ULL << offset_)
     {
       FUSE_INIT_FLAG_CASE(ASYNC_READ);
       FUSE_INIT_FLAG_CASE(POSIX_LOCKS);
@@ -189,6 +189,17 @@ fuse_flag_to_str(const uint32_t offset_)
       FUSE_INIT_FLAG_CASE(NO_OPENDIR_SUPPORT);
       FUSE_INIT_FLAG_CASE(EXPLICIT_INVAL_DATA);
       FUSE_INIT_FLAG_CASE(MAP_ALIGNMENT);
+      FUSE_INIT_FLAG_CASE(SUBMOUNTS);
+      FUSE_INIT_FLAG_CASE(HANDLE_KILLPRIV_V2);
+      FUSE_INIT_FLAG_CASE(SETXATTR_EXT);
+      FUSE_INIT_FLAG_CASE(INIT_EXT);
+      FUSE_INIT_FLAG_CASE(INIT_RESERVED);
+      FUSE_INIT_FLAG_CASE(SECURITY_CTX);
+      FUSE_INIT_FLAG_CASE(HAS_INODE_DAX);
+      FUSE_INIT_FLAG_CASE(CREATE_SUPP_GROUP);
+      FUSE_INIT_FLAG_CASE(HAS_EXPIRE_ONLY);
+      FUSE_INIT_FLAG_CASE(DIRECT_IO_ALLOW_MMAP);
+      FUSE_INIT_FLAG_CASE(PASSTHROUGH);
     }
 
   return NULL;
@@ -200,7 +211,7 @@ static
 void
 debug_open_flags(const uint32_t flags_)
 {
-  fprintf(stderr,"%s,",open_accmode_to_str(flags_));
+  fprintf(stderr,"%s, ",open_accmode_to_str(flags_));
   for(int i = 0; i < (sizeof(flags_) * 8); i++)
     {
       const char *str;
@@ -212,7 +223,7 @@ debug_open_flags(const uint32_t flags_)
       if(str == NULL)
         continue;
 
-      fprintf(stderr,"%s,",str);
+      fprintf(stderr,"%s, ",str);
     }
 }
 
@@ -717,28 +728,31 @@ debug_fuse_fallocate_in(const void *arg_)
 void
 debug_fuse_init_in(const struct fuse_init_in *arg_)
 {
+  uint64_t flags;
+
+  flags = (((uint64_t)arg_->flags) | ((uint64_t)arg_->flags2) << 32);
   fprintf(g_OUTPUT,
           "FUSE_INIT_IN: "
           " major=%u;"
           " minor=%u;"
           " max_readahead=%u;"
-          " flags=0x%08X (",
+          " flags=0x%016lx (",
           arg_->major,
           arg_->minor,
           arg_->max_readahead,
-          arg_->flags);
-  for(uint64_t i = 0; i < (sizeof(arg_->flags)*8); i++)
+          flags);
+  for(uint64_t i = 0; i < (sizeof(flags)*8); i++)
     {
       const char *str;
 
-      if(!(arg_->flags & (1ULL << i)))
+      if(!(flags & (1ULL << i)))
         continue;
 
       str = fuse_flag_to_str(i);
       if(str == NULL)
         continue;
 
-      fprintf(g_OUTPUT,"%s,",str);
+      fprintf(g_OUTPUT,"%s, ",str);
     }
   fprintf(g_OUTPUT,")\n");
 }
@@ -748,7 +762,10 @@ debug_fuse_init_out(const uint64_t              unique_,
                     const struct fuse_init_out *arg_,
                     const uint64_t              argsize_)
 {
+  uint64_t flags;
   const struct fuse_init_out *arg = arg_;
+
+  flags = (((uint64_t)arg->flags) | ((uint64_t)arg->flags2) << 32);
   fprintf(g_OUTPUT,
           /* "unique=0x%016"PRIx64";" */
           /* " opcode=RESPONSE;" */
@@ -758,27 +775,27 @@ debug_fuse_init_out(const uint64_t              unique_,
           " major=%u;"
           " minor=%u;"
           " max_readahead=%u;"
-          " flags=0x%08X ("
+          " flags=0x%016lx ("
           ,
           /* unique_, */
           /* sizeof(struct fuse_out_header) + argsize_, */
           arg->major,
           arg->minor,
           arg->max_readahead,
-          arg->flags);
+          flags);
 
-  for(uint64_t i = 0; i < (sizeof(arg->flags)*8); i++)
+  for(uint64_t i = 0; i < (sizeof(flags)*8); i++)
     {
       const char *str;
 
-      if(!(arg->flags & (1ULL << i)))
+      if(!(flags & (1ULL << i)))
         continue;
 
       str = fuse_flag_to_str(i);
       if(str == NULL)
         continue;
 
-      fprintf(g_OUTPUT,"%s,",str);
+      fprintf(g_OUTPUT,"%s, ",str);
     }
 
   fprintf(g_OUTPUT,
@@ -788,13 +805,15 @@ debug_fuse_init_out(const uint64_t              unique_,
           " time_gran=%u;"
           " max_pages=%u;"
           " map_alignment=%u;"
+          " max_stack_depth=%u;"
           "\n",
           arg->max_background,
           arg->congestion_threshold,
           arg->max_write,
           arg->time_gran,
           arg->max_pages,
-          arg->map_alignment);
+          arg->map_alignment,
+          arg->max_stack_depth);
 }
 
 static

libfuse/lib/fuse.c

@@ -47,6 +47,7 @@
 #include <syslog.h>
 #include <time.h>
 #include <unistd.h>
+#include <sys/ioctl.h>
 
 #ifdef HAVE_MALLOC_TRIM
 #include <malloc.h>
@@ -1375,7 +1376,7 @@ set_path_info(struct fuse             *f,
     return -ENOMEM;
 
   e->ino        = node->nodeid;
-  e->generation = f->nodeid_gen.generation;
+  e->generation = ((e->ino == FUSE_ROOT_ID) ? 0 : f->nodeid_gen.generation);
 
   pthread_mutex_lock(&f->lock);
   update_stat(node,&e->attr);
@@ -1386,6 +1387,15 @@ set_path_info(struct fuse             *f,
   return 0;
 }
 
+/*
+  lookup requests only come in for FUSE_ROOT_ID when a "parent of
+  child of root node" request is made. This can happen when using
+  EXPORT_SUPPORT=true and a file handle is used to keep a reference to
+  a node which has been forgotten. Mostly a NFS concern but not
+  excluslively. Root node always has a nodeid of 1 and generation of
+  0. To ensure this set_path_info() explicitly ensures the root id has
+  a generation of 0.
+ */
 static
 int
 lookup_path(struct fuse             *f,
@@ -3673,6 +3683,12 @@ fuse_get_context(void)
   return &fuse_get_context_internal()->ctx;
 }
 
+int
+fuse_get_dev_fuse_fd(const struct fuse_context *fc_)
+{
+  return fuse_chan_fd(fc_->fuse->se->ch);
+}
+
 enum {
       KEY_HELP,
 };
@@ -4147,3 +4163,30 @@ fuse_log_metrics_get(void)
 {
   return g_LOG_METRICS;
 }
+
+int
+fuse_passthrough_open(const struct fuse_context *fc_,
+                      const int                  fd_)
+{
+  int rv;
+  int dev_fuse_fd;
+  struct fuse_backing_map bm = {0};
+
+  dev_fuse_fd = fuse_get_dev_fuse_fd(fc_);
+  bm.fd       = fd_;
+
+  rv = ioctl(dev_fuse_fd,FUSE_DEV_IOC_BACKING_OPEN,&bm);
+
+  return rv;
+}
+
+int
+fuse_passthrough_close(const struct fuse_context *fc_,
+                       const int                  backing_id_)
+{
+  int dev_fuse_fd;
+
+  dev_fuse_fd = fuse_get_dev_fuse_fd(fc_);
+
+  return ioctl(dev_fuse_fd, FUSE_DEV_IOC_BACKING_CLOSE, &backing_id_);
+}

libfuse/lib/fuse_lowlevel.c

@@ -277,6 +277,11 @@ fill_open(struct fuse_open_out   *arg_,
     arg_->open_flags |= FOPEN_PARALLEL_DIRECT_WRITES;
   if(ffi_->noflush)
     arg_->open_flags |= FOPEN_NOFLUSH;
+  if(ffi_->passthrough)
+    {
+      arg_->open_flags |= FOPEN_PASSTHROUGH;
+      arg_->backing_id = ffi_->backing_id;
+    }
 }
 
 int
@@ -294,6 +299,11 @@ fuse_reply_entry(fuse_req_t                     req,
 
   fill_entry(&arg, e);
 
+#ifdef NDEBUG
+  // TODO: Add checks for cases where a node could be marked bad by
+  // the kernel.
+#endif
+
   return send_reply_ok(req, &arg, size);
 }
 
@@ -1092,6 +1102,11 @@ do_init(fuse_req_t             req,
   struct fuse_init_in *arg = (struct fuse_init_in *) &hdr_[1];
   struct fuse_ll *f = req->f;
   size_t bufsize = fuse_chan_bufsize(req->ch);
+  uint64_t inargflags;
+  uint64_t outargflags;
+
+  inargflags = 0;
+  outargflags = 0;
 
   if(f->debug)
     debug_fuse_init_in(arg);
@@ -1122,40 +1137,55 @@ do_init(fuse_req_t             req,
 
   if(arg->minor >= 6)
     {
+      inargflags = arg->flags;
+      if(inargflags & FUSE_INIT_EXT)
+        inargflags |= (((uint64_t)arg->flags2) << 32);
+
       if(arg->max_readahead < f->conn.max_readahead)
         f->conn.max_readahead = arg->max_readahead;
-      if(arg->flags & FUSE_ASYNC_READ)
+
+      if(inargflags & FUSE_ASYNC_READ)
         f->conn.capable |= FUSE_CAP_ASYNC_READ;
-      if(arg->flags & FUSE_POSIX_LOCKS)
+      if(inargflags & FUSE_POSIX_LOCKS)
         f->conn.capable |= FUSE_CAP_POSIX_LOCKS;
-      if(arg->flags & FUSE_ATOMIC_O_TRUNC)
+      if(inargflags & FUSE_ATOMIC_O_TRUNC)
         f->conn.capable |= FUSE_CAP_ATOMIC_O_TRUNC;
-      if(arg->flags & FUSE_EXPORT_SUPPORT)
+      if(inargflags & FUSE_EXPORT_SUPPORT)
         f->conn.capable |= FUSE_CAP_EXPORT_SUPPORT;
-      if(arg->flags & FUSE_BIG_WRITES)
+      if(inargflags & FUSE_BIG_WRITES)
         f->conn.capable |= FUSE_CAP_BIG_WRITES;
-      if(arg->flags & FUSE_DONT_MASK)
+      if(inargflags & FUSE_DONT_MASK)
         f->conn.capable |= FUSE_CAP_DONT_MASK;
-      if(arg->flags & FUSE_FLOCK_LOCKS)
+      if(inargflags & FUSE_FLOCK_LOCKS)
         f->conn.capable |= FUSE_CAP_FLOCK_LOCKS;
-      if(arg->flags & FUSE_POSIX_ACL)
+      if(inargflags & FUSE_POSIX_ACL)
         f->conn.capable |= FUSE_CAP_POSIX_ACL;
-      if(arg->flags & FUSE_CACHE_SYMLINKS)
+      if(inargflags & FUSE_CACHE_SYMLINKS)
         f->conn.capable |= FUSE_CAP_CACHE_SYMLINKS;
-      if(arg->flags & FUSE_ASYNC_DIO)
+      if(inargflags & FUSE_ASYNC_DIO)
         f->conn.capable |= FUSE_CAP_ASYNC_DIO;
-      if(arg->flags & FUSE_PARALLEL_DIROPS)
+      if(inargflags & FUSE_PARALLEL_DIROPS)
         f->conn.capable |= FUSE_CAP_PARALLEL_DIROPS;
-      if(arg->flags & FUSE_MAX_PAGES)
+      if(inargflags & FUSE_MAX_PAGES)
         f->conn.capable |= FUSE_CAP_MAX_PAGES;
-      if(arg->flags & FUSE_WRITEBACK_CACHE)
+      if(inargflags & FUSE_WRITEBACK_CACHE)
         f->conn.capable |= FUSE_CAP_WRITEBACK_CACHE;
-      if(arg->flags & FUSE_DO_READDIRPLUS)
+      if(inargflags & FUSE_DO_READDIRPLUS)
         f->conn.capable |= FUSE_CAP_READDIR_PLUS;
-      if(arg->flags & FUSE_READDIRPLUS_AUTO)
+      if(inargflags & FUSE_READDIRPLUS_AUTO)
         f->conn.capable |= FUSE_CAP_READDIR_PLUS_AUTO;
-      if(arg->flags & FUSE_SETXATTR_EXT)
+      if(inargflags & FUSE_SETXATTR_EXT)
         f->conn.capable |= FUSE_CAP_SETXATTR_EXT;
+      if(inargflags & FUSE_DIRECT_IO_ALLOW_MMAP)
+        f->conn.capable |= FUSE_CAP_DIRECT_IO_ALLOW_MMAP;
+      if(inargflags & FUSE_CREATE_SUPP_GROUP)
+        f->conn.capable |= FUSE_CAP_CREATE_SUPP_GROUP;
+      if(inargflags & FUSE_PASSTHROUGH)
+        f->conn.capable |= FUSE_CAP_PASSTHROUGH;
+      if(inargflags & FUSE_HANDLE_KILLPRIV)
+        f->conn.capable |= FUSE_CAP_HANDLE_KILLPRIV;
+      if(inargflags & FUSE_HANDLE_KILLPRIV_V2)
+        f->conn.capable |= FUSE_CAP_HANDLE_KILLPRIV_V2;
     }
   else
     {
@@ -1186,44 +1216,66 @@ do_init(fuse_req_t             req,
   if(f->op.init)
     f->op.init(f->userdata, &f->conn);
 
-  if((arg->flags & FUSE_MAX_PAGES) && (f->conn.want & FUSE_CAP_MAX_PAGES))
+  outargflags = outarg.flags;
+  if((inargflags & FUSE_MAX_PAGES) && (f->conn.want & FUSE_CAP_MAX_PAGES))
     {
-      outarg.flags     |= FUSE_MAX_PAGES;
+      outargflags     |= FUSE_MAX_PAGES;
       outarg.max_pages  = f->conn.max_pages;
 
       msgbuf_set_bufsize(outarg.max_pages + 1);
     }
 
   if(f->conn.want & FUSE_CAP_ASYNC_READ)
-    outarg.flags |= FUSE_ASYNC_READ;
+    outargflags |= FUSE_ASYNC_READ;
   if(f->conn.want & FUSE_CAP_POSIX_LOCKS)
-    outarg.flags |= FUSE_POSIX_LOCKS;
+    outargflags |= FUSE_POSIX_LOCKS;
   if(f->conn.want & FUSE_CAP_ATOMIC_O_TRUNC)
-    outarg.flags |= FUSE_ATOMIC_O_TRUNC;
+    outargflags |= FUSE_ATOMIC_O_TRUNC;
   if(f->conn.want & FUSE_CAP_EXPORT_SUPPORT)
-    outarg.flags |= FUSE_EXPORT_SUPPORT;
+    outargflags |= FUSE_EXPORT_SUPPORT;
   if(f->conn.want & FUSE_CAP_BIG_WRITES)
-    outarg.flags |= FUSE_BIG_WRITES;
+    outargflags |= FUSE_BIG_WRITES;
   if(f->conn.want & FUSE_CAP_DONT_MASK)
-    outarg.flags |= FUSE_DONT_MASK;
+    outargflags |= FUSE_DONT_MASK;
   if(f->conn.want & FUSE_CAP_FLOCK_LOCKS)
-    outarg.flags |= FUSE_FLOCK_LOCKS;
+    outargflags |= FUSE_FLOCK_LOCKS;
   if(f->conn.want & FUSE_CAP_POSIX_ACL)
-    outarg.flags |= FUSE_POSIX_ACL;
+    outargflags |= FUSE_POSIX_ACL;
   if(f->conn.want & FUSE_CAP_CACHE_SYMLINKS)
-    outarg.flags |= FUSE_CACHE_SYMLINKS;
+    outargflags |= FUSE_CACHE_SYMLINKS;
   if(f->conn.want & FUSE_CAP_ASYNC_DIO)
-    outarg.flags |= FUSE_ASYNC_DIO;
+    outargflags |= FUSE_ASYNC_DIO;
   if(f->conn.want & FUSE_CAP_PARALLEL_DIROPS)
-    outarg.flags |= FUSE_PARALLEL_DIROPS;
+    outargflags |= FUSE_PARALLEL_DIROPS;
   if(f->conn.want & FUSE_CAP_WRITEBACK_CACHE)
-    outarg.flags |= FUSE_WRITEBACK_CACHE;
+    outargflags |= FUSE_WRITEBACK_CACHE;
   if(f->conn.want & FUSE_CAP_READDIR_PLUS)
-    outarg.flags |= FUSE_DO_READDIRPLUS;
+    outargflags |= FUSE_DO_READDIRPLUS;
   if(f->conn.want & FUSE_CAP_READDIR_PLUS_AUTO)
-    outarg.flags |= FUSE_READDIRPLUS_AUTO;
+    outargflags |= FUSE_READDIRPLUS_AUTO;
   if(f->conn.want & FUSE_CAP_SETXATTR_EXT)
-    outarg.flags |= FUSE_SETXATTR_EXT;
+    outargflags |= FUSE_SETXATTR_EXT;
+  if(f->conn.want & FUSE_CAP_CREATE_SUPP_GROUP)
+    outargflags |= FUSE_CREATE_SUPP_GROUP;
+  if(f->conn.want & FUSE_CAP_DIRECT_IO_ALLOW_MMAP)
+    outargflags |= FUSE_DIRECT_IO_ALLOW_MMAP;
+  if(f->conn.want & FUSE_CAP_HANDLE_KILLPRIV)
+    outargflags |= FUSE_HANDLE_KILLPRIV;
+  if(f->conn.want & FUSE_CAP_HANDLE_KILLPRIV_V2)
+    outargflags |= FUSE_HANDLE_KILLPRIV_V2;
+  if(f->conn.want & FUSE_CAP_PASSTHROUGH)
+    {
+      outargflags |= FUSE_PASSTHROUGH;
+      outarg.max_stack_depth = 2;
+    }
+
+  if(inargflags & FUSE_INIT_EXT)
+    {
+      outargflags |= FUSE_INIT_EXT;
+      outarg.flags2 = (outargflags >> 32);
+    }
+
+  outarg.flags = outargflags;
 
   outarg.max_readahead = f->conn.max_readahead;
   outarg.max_write = f->conn.max_write;
@@ -1242,6 +1294,9 @@ do_init(fuse_req_t             req,
       outarg.congestion_threshold = f->conn.congestion_threshold;
     }
 
+  if(f->conn.proto_minor >= 23)
+    outarg.time_gran = 1;
+
   size_t outargsize;
   if(arg->minor < 5)
     outargsize = FUSE_COMPAT_INIT_OUT_SIZE;

man/mergerfs.1

@@ -106,16 +106,22 @@ If you don\[cq]t already know that you have a special use case then just
 start with one of the following option sets.
 .SS You need \f[C]mmap\f[R] (used by rtorrent and many sqlite3 base software)
 .PP
-\f[C]cache.files=partial,dropcacheonclose=true,category.create=mfs\f[R]
+\f[C]cache.files=auto-full,dropcacheonclose=true,category.create=mfs\f[R]
+.PP
+or if you are on a Linux kernel >= 6.6.x mergerfs will enable a mode
+that allows shared mmap when \f[C]cache.files=off\f[R].
+To be sure of the best performance between \f[C]cache.files=off\f[R] and
+\f[C]cache.files=auto-full\f[R] you\[cq]ll need to do your own
+benchmarking but often \f[C]off\f[R] is faster.
 .SS You don\[cq]t need \f[C]mmap\f[R]
 .PP
 \f[C]cache.files=off,dropcacheonclose=true,category.create=mfs\f[R]
 .SS Command Line
 .PP
-\f[C]mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media\f[R]
+\f[C]mergerfs -o cache.files=auto-full,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media\f[R]
 .SS /etc/fstab
 .PP
-\f[C]/mnt/hdd0:/mnt/hdd1 /media fuse.mergerfs cache.files=partial,dropcacheonclose=true,category.create=mfs 0 0\f[R]
+\f[C]/mnt/hdd0:/mnt/hdd1 /media mergerfs cache.files=auto-full,dropcacheonclose=true,category.create=mfs 0 0\f[R]
 .SS systemd mount
 .PP
 https://github.com/trapexit/mergerfs/wiki/systemd
@@ -130,7 +136,7 @@ Type=simple
 KillMode=none
 ExecStart=/usr/bin/mergerfs \[rs]
   -f \[rs]
-  -o cache.files=partial \[rs]
+  -o cache.files=auto-full \[rs]
   -o dropcacheonclose=true \[rs]
   -o category.create=mfs \[rs]
   /mnt/hdd0:/mnt/hdd1 \[rs]
@@ -185,6 +191,13 @@ Recommended when
 caching.
 (default: false)
 .IP \[bu] 2
+\f[B]direct-io-allow-mmap=BOOL\f[R]: On newer kernels (>= 6.6) it is
+possible to disable file page caching while still allowing for shared
+mmap support.
+mergerfs will enable this feature if available but an option is provided
+to turn it off for testing and debugging purposes.
+(default: true)
+.IP \[bu] 2
 \f[B]symlinkify=BOOL\f[R]: When enabled and a file is not writable and
 its mtime or ctime is older than \f[B]symlinkify_timeout\f[R] files will
 be reported as symlinks to the original files.
@@ -516,8 +529,8 @@ tools use \f[B]/etc/fstab\f[R].
 .IP
 .nf
 \f[C]
-# <file system>        <mount point>  <type>         <options>             <dump>  <pass>
-/mnt/hdd*:/mnt/ssd    /media          fuse.mergerfs  minfreespace=16G      0       0
+# <file system>        <mount point>  <type>    <options>             <dump>  <pass>
+/mnt/hdd*:/mnt/ssd    /media          mergerfs  minfreespace=16G      0       0
 \f[R]
 .fi
 .PP
@@ -847,20 +860,12 @@ In theory this flag should not be exposed to the end user.
 It is a low-level FUSE flag which indicates whether or not the kernel
 can send certain kinds of messages to it for the purposes of using with
 NFS.
-mergerfs does support these messages but due to a possible bug in the
-Linux kernel this option needs to be configurable.
-As it turns out at least certain versions of NFS can still work when
-disabled and no longer triggers the bug.
-.PP
-While it may be safe to default this setting to \f[C]false\f[R] in an
-abundance of caution it is being left enabled by default.
+mergerfs does support these messages but due bugs and quirks found in
+the kernel and mergerfs this option is provided just in case it is
+needed for debugging.
 .PP
 Given that this flag is set when the FUSE connection is first initiated
 it is not possible to change during run time.
-.PP
-See Kernel Issues &
-Bugs (https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs)
-for more details.
 .SH FUNCTIONS, CATEGORIES and POLICIES
 .PP
 The POSIX filesystem API is made up of a number of functions.
@@ -1577,8 +1582,8 @@ apply those before mounting the new version.
 \f[C]
 $ sudo mount /mnt/mergerfs
 $ mount | grep mergerfs
-media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
-media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
+media on /mnt/mergerfs type mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
+media on /mnt/mergerfs type mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
 \f[R]
 .fi
 .PP
@@ -2404,8 +2409,8 @@ Take a look at the section on NFS in the #remote-filesystems for more
 details.
 .SS rtorrent fails with ENODEV (No such device)
 .PP
-Be sure to set \f[C]cache.files=partial|full|auto-full|per-processe\f[R]
-or turn off \f[C]direct_io\f[R].
+Be sure to set
+\f[C]cache.files=partial|full|auto-full|per-processe\f[R].
 rtorrent and some other applications use
 mmap (http://linux.die.net/man/2/mmap) to read and write to files and
 offer no fallback to traditional methods.
@@ -2588,11 +2593,13 @@ It should be noted that NFS and FUSE (the technology mergerfs uses) do
 not work perfectly with one another due to certain design choices in
 FUSE (and mergerfs.) Due to these issues it is generally recommended to
 use SMB when possible till situations change.
-That said issues should still be reported.
-NFS is not really recommended but it isn\[cq]t unsupported.
+That said mergerfs should generally work as an export of NFS and issues
+discovered should still be reported.
 .PP
-mergerfs settings: * noforget * inodecalc=path-hash *
-export-support=false (only available in v2.40.0 and above)
+To ensure compatibility between mergerfs and NFS use the following
+settings.
+.PP
+mergerfs settings: * noforget * inodecalc=path-hash
 .PP
 NFS export settings: * fsid=UUID * no_root_squash
 .PP
@@ -2618,35 +2625,17 @@ mergerfs will use will be on a different branch and therefore the inode
 would change.
 This isn\[cq]t an ideal solution and others are being considered but it
 works for most situations.
-It might be possible to leave the default value of \f[C]hybrid-hash\f[R]
-if \f[C]export-support=false\f[R] but it hasn\[cq]t been fully tested
-yet.
-.PP
-\f[C]export-support=false\f[R] is a low-level FUSE option that is needed
-as a workaround to a Linux kernel bug.
-Technically this option should be set to \f[C]true\f[R] when expecting
-to use mergerfs with NFS but it turns out that at least some versions of
-NFS work when not enabled.
-At the moment it is unclear if this has any negative side effects but
-when disabled NFS appears to work alright and the kernel no longer sends
-invalid requests.
-When the issue is resolved in the kernel these docs will be updated with
-appropriate details.
-.PP
-Also see Kernel Issues &
-Bugs (https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs)
-for more details.
 .PP
 \f[C]fsid=UUID\f[R] is needed because FUSE filesystems don\[cq]t have
 different \f[C]st_dev\f[R] values which can cause issues when exporting.
 The easiest thing to do is set each mergerfs export \f[C]fsid\f[R] to
 some random value.
 An easy way to generate a random value is to use the command line tool
-\f[C]uuidgen\f[R] or through a website such as
+\f[C]uuid\f[R] or \f[C]uuidgen\f[R] or through a website such as
 uuidgenerator.net (https://www.uuidgenerator.net/).
 .PP
 \f[C]no_root_squash\f[R] is not strictly necessary but can lead to
-confusing permission and ownership issues.
+confusing permission and ownership issues if root squashing is enabled.
 .SS SMB / CIFS
 .PP
 SMB (https://en.wikipedia.org/wiki/Server_Message_Block) is a protocol

src/config.cpp

@@ -58,9 +58,12 @@ namespace l
     IFERT("branches-mount-timeout");
     IFERT("cache.symlinks");
     IFERT("cache.writeback");
+    IFERT("direct-io-allow-mmap");
     IFERT("export-support");
     IFERT("fsname");
     IFERT("fuse_msg_size");
+    IFERT("handle-killpriv");
+    IFERT("handle-killpriv-v2");
     IFERT("mount");
     IFERT("nullrw");
     IFERT("pid");
@@ -94,6 +97,7 @@ Config::Config()
     cache_symlinks(false),
     category(func),
     direct_io(false),
+    direct_io_allow_mmap(true),
     dropcacheonclose(false),
     export_support(true),
     flushonclose(FlushOnClose::ENUM::OPENED_FOR_WRITE),
@@ -101,6 +105,8 @@ Config::Config()
     fsname(),
     func(),
     fuse_msg_size(FUSE_MAX_MAX_PAGES),
+    handle_killpriv(true),
+    handle_killpriv_v2(true),
     ignorepponrename(false),
     inodecalc("hybrid-hash"),
     lazy_umount_mountpoint(false),
@@ -112,6 +118,7 @@ Config::Config()
     nfsopenhack(NFSOpenHack::ENUM::OFF),
     nullrw(false),
     parallel_direct_writes(false),
+    passthrough(PassthroughEnum::off),
     posix_acl(false),
     readahead(0),
     readdir("seq"),
@@ -150,6 +157,7 @@ Config::Config()
   _map["category.create"]        = &category.create;
   _map["category.search"]        = &category.search;
   _map["direct_io"]              = &direct_io;
+  _map["direct-io-allow-mmap"]   = &direct_io_allow_mmap;
   _map["dropcacheonclose"]       = &dropcacheonclose;
   _map["export-support"]         = &export_support;
   _map["flush-on-close"]         = &flushonclose;
@@ -177,6 +185,8 @@ Config::Config()
   _map["func.unlink"]            = &func.unlink;
   _map["func.utimens"]           = &func.utimens;
   _map["fuse_msg_size"]          = &fuse_msg_size;
+  _map["handle-killpriv"]        = &handle_killpriv;
+  _map["handle-killpriv-v2"]     = &handle_killpriv_v2;
   _map["ignorepponrename"]       = &ignorepponrename;
   _map["inodecalc"]              = &inodecalc;
   _map["kernel_cache"]           = &kernel_cache;
@@ -191,6 +201,7 @@ Config::Config()
   _map["nullrw"]                 = &nullrw;
   _map["pid"]                    = &pid;
   _map["parallel-direct-writes"] = &parallel_direct_writes;
+  _map["passthrough"]            = &passthrough;
   _map["pin-threads"]            = &fuse_pin_threads;
   _map["posix_acl"]              = &posix_acl;
   _map["readahead"]              = &readahead;

src/config.hpp

@@ -21,12 +21,13 @@
 #include "config_cachefiles.hpp"
 #include "config_flushonclose.hpp"
 #include "config_follow_symlinks.hpp"
-#include "config_pid.hpp"
 #include "config_inodecalc.hpp"
 #include "config_link_exdev.hpp"
 #include "config_log_metrics.hpp"
 #include "config_moveonenospc.hpp"
 #include "config_nfsopenhack.hpp"
+#include "config_passthrough.hpp"
+#include "config_pid.hpp"
 #include "config_rename_exdev.hpp"
 #include "config_set.hpp"
 #include "config_statfs.hpp"
@@ -117,6 +118,7 @@ public:
   ConfigBOOL     cache_symlinks;
   Categories     category;
   ConfigBOOL     direct_io;
+  ConfigBOOL     direct_io_allow_mmap;
   ConfigBOOL     dropcacheonclose;
   ConfigBOOL     export_support;
   FlushOnClose   flushonclose;
@@ -124,6 +126,8 @@ public:
   ConfigSTR      fsname;
   Funcs          func;
   ConfigUINT64   fuse_msg_size;
+  ConfigBOOL     handle_killpriv;
+  ConfigBOOL     handle_killpriv_v2;
   ConfigBOOL     ignorepponrename;
   InodeCalc      inodecalc;
   ConfigBOOL     kernel_cache;
@@ -136,6 +140,7 @@ public:
   NFSOpenHack    nfsopenhack;
   ConfigBOOL     nullrw;
   ConfigBOOL     parallel_direct_writes;
+  Passthrough    passthrough;
   ConfigGetPid   pid;
   ConfigBOOL     posix_acl;
   ConfigUINT64   readahead;

src/config_passthrough.cpp

@@ -0,0 +1,43 @@
+#/*
+  ISC License
+
+  Copyright (c) 2024, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#include "config_passthrough.hpp"
+#include "ef.hpp"
+#include "errno.hpp"
+
+template<>
+std::string
+Passthrough::to_string() const
+{
+  return _data._to_string();
+}
+
+template<>
+int
+Passthrough::from_string(const std::string &s_)
+{
+  better_enums::optional<PassthroughEnum> e;
+
+  e = PassthroughEnum::_from_string_nothrow(s_.c_str());
+  if(!e)
+    return -EINVAL;
+  
+  _data = *e;
+
+  return 0;
+}

src/config_passthrough.hpp

@@ -0,0 +1,28 @@
+/*
+  ISC License
+
+  Copyright (c) 2024, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#pragma once
+
+#include "better_enum.hpp"
+
+#include "enum.hpp"
+
+
+BETTER_ENUM(PassthroughEnum, uint8_t, off, ro, wo, rw);
+
+typedef Enum<PassthroughEnum> Passthrough;

src/enum.hpp

@@ -60,6 +60,12 @@ public:
     return _data;
   }
 
+  ENUM
+  operator+() const
+  {
+    return _data;
+  }
+
 public:
   bool operator==(const ENUM data_) const
   {

src/fileinfo.hpp

@@ -31,12 +31,14 @@ public:
            bool const  direct_io_)
     : FH(fusepath_),
       fd(fd_),
+      backing_id(0),
       direct_io(direct_io_)
   {
   }
 
 public:
   int fd;
+  int backing_id;
   uint32_t direct_io:1;
   std::mutex mutex;
 };

src/fuse_create.cpp

@@ -170,6 +170,29 @@ namespace l
     return 0;
   }
 
+  static
+  int
+  passthrough(const fuse_context *fc_,
+              fuse_file_info_t   *ffi_)
+  {
+    int backing_id;
+    FileInfo *fi;
+    const ugid::SetRootGuard ugid;
+
+    fi = reinterpret_cast<FileInfo*>(ffi_->fh);
+
+    backing_id = fuse_passthrough_open(fc_,fi->fd);
+    if(backing_id <= 0)
+      return 0;
+
+    ffi_->passthrough = true;
+    ffi_->keep_cache  = false;
+    fi->backing_id    = backing_id;
+    ffi_->backing_id  = backing_id;
+
+    return 0;
+  }
+
   static
   int
   create(const Policy::Search &searchFunc_,
@@ -208,6 +231,15 @@ namespace l
   }
 }
 
+constexpr
+const
+uint64_t
+_(const PassthroughEnum e,
+  const uint64_t        m)
+{
+  return ((((uint64_t)e) << 32) | (m & O_ACCMODE));
+}
+
 namespace FUSE
 {
   int
@@ -247,6 +279,17 @@ namespace FUSE
                        fc->umask);
       }
 
+    if(rv != 0)
+      return rv;
+
+    switch(_(cfg->passthrough,ffi_->flags))
+      {
+      case _(PassthroughEnum::ro,O_RDONLY):
+      case _(PassthroughEnum::wo,O_WRONLY):
+      case _(PassthroughEnum::rw,O_RDWR):
+        return l::passthrough(fc,ffi_);
+      }
+
     return rv;
   }
 }

src/fuse_init.cpp

@@ -144,14 +144,18 @@ namespace FUSE
     l::want_if_capable(conn_,FUSE_CAP_ATOMIC_O_TRUNC);
     l::want_if_capable(conn_,FUSE_CAP_BIG_WRITES);
     l::want_if_capable(conn_,FUSE_CAP_CACHE_SYMLINKS,&cfg->cache_symlinks);
+    l::want_if_capable(conn_,FUSE_CAP_DIRECT_IO_ALLOW_MMAP,&cfg->direct_io_allow_mmap);
     l::want_if_capable(conn_,FUSE_CAP_DONT_MASK);
     l::want_if_capable(conn_,FUSE_CAP_EXPORT_SUPPORT,&cfg->export_support);
+    l::want_if_capable(conn_,FUSE_CAP_HANDLE_KILLPRIV,&cfg->handle_killpriv);
+    l::want_if_capable(conn_,FUSE_CAP_HANDLE_KILLPRIV_V2,&cfg->handle_killpriv_v2);
     l::want_if_capable(conn_,FUSE_CAP_IOCTL_DIR);
     l::want_if_capable(conn_,FUSE_CAP_PARALLEL_DIROPS);
+    l::want_if_capable(conn_,FUSE_CAP_PASSTHROUGH);
     l::want_if_capable(conn_,FUSE_CAP_POSIX_ACL,&cfg->posix_acl);
     l::want_if_capable(conn_,FUSE_CAP_READDIR_PLUS,&cfg->readdirplus);
-    //    l::want_if_capable(conn_,FUSE_CAP_READDIR_PLUS_AUTO);
     l::want_if_capable(conn_,FUSE_CAP_WRITEBACK_CACHE,&cfg->writeback_cache);
+    //    l::want_if_capable(conn_,FUSE_CAP_READDIR_PLUS_AUTO);
     l::want_if_capable_max_pages(conn_,cfg);
     conn_->want &= ~FUSE_CAP_POSIX_LOCKS;
     conn_->want &= ~FUSE_CAP_FLOCK_LOCKS;

src/fuse_open.cpp

@@ -218,6 +218,29 @@ namespace l
     return 0;
   }
 
+  static
+  int
+  passthrough(const fuse_context *fc_,
+              fuse_file_info_t   *ffi_)
+  {
+    int backing_id;
+    FileInfo *fi;
+    const ugid::SetRootGuard ugid;
+
+    fi = reinterpret_cast<FileInfo*>(ffi_->fh);
+
+    backing_id = fuse_passthrough_open(fc_,fi->fd);
+    if(backing_id <= 0)
+      return 0;
+
+    ffi_->passthrough = true;
+    ffi_->keep_cache  = false;
+    fi->backing_id    = backing_id;
+    ffi_->backing_id  = backing_id;
+
+    return 0;
+  }
+
   static
   int
   open(const Policy::Search &searchFunc_,
@@ -238,6 +261,15 @@ namespace l
   }
 }
 
+constexpr
+const
+uint64_t
+_(const PassthroughEnum e,
+  const uint64_t        m)
+{
+  return ((((uint64_t)e) << 32) | (m & O_ACCMODE));
+}
+
 namespace FUSE
 {
   int
@@ -264,6 +296,17 @@ namespace FUSE
                  cfg->link_cow,
                  cfg->nfsopenhack);
 
+    if(rv != 0)
+      return rv;
+
+    switch(_(cfg->passthrough,ffi_->flags))
+      {
+      case _(PassthroughEnum::ro,O_RDONLY):
+      case _(PassthroughEnum::wo,O_WRONLY):
+      case _(PassthroughEnum::rw,O_RDWR):
+        return l::passthrough(fc,ffi_);
+      }
+
     return rv;
   }
 }

src/fuse_release.cpp

@@ -19,6 +19,7 @@
 #include "fileinfo.hpp"
 #include "fs_close.hpp"
 #include "fs_fadvise.hpp"
+#include "ugid.hpp"
 
 #include "fuse.h"
 
@@ -40,6 +41,16 @@ namespace l
         fs::fadvise_dontneed(fi_->fd);
       }
 
+    if(fi_->backing_id)
+      {
+        const fuse_context *fc;
+        const ugid::SetRootGuard ugid;
+
+        fc = fuse_get_context();
+
+        fuse_passthrough_close(fc,fi_->backing_id);
+      }
+
     fs::close(fi_->fd);
 
     delete fi_;

tools/preload.c

@@ -107,7 +107,7 @@ open(const char *pathname_,
   LOAD_FUNC(open);
 
   mode = 0;
-  if(flags_ & O_CREAT)
+  if(flags_ & (O_CREAT|O_TMPFILE))
     {
       va_list args;
       va_start(args,flags_);
@@ -119,7 +119,7 @@ open(const char *pathname_,
   if(fd == -1)
     return -1;
 
-  if(flags_ & (O_DIRECTORY|O_PATH))
+  if(flags_ & (O_DIRECTORY|O_PATH|O_TMPFILE))
     return fd;
   rv = fstat(fd,&st);
   if(rv == -1)
@@ -155,7 +155,7 @@ open64(const char *pathname_,
   LOAD_FUNC(open64);
 
   mode = 0;
-  if(flags_ & O_CREAT)
+  if(flags_ & (O_CREAT|O_TMPFILE))
     {
       va_list args;
       va_start(args,flags_);
@@ -167,7 +167,7 @@ open64(const char *pathname_,
   if(fd == -1)
     return -1;
 
-  if(flags_ & (O_DIRECTORY|O_PATH))
+  if(flags_ & (O_DIRECTORY|O_PATH|O_TMPFILE))
     return fd;
   rv = fstat(fd,&st);
   if(rv == -1)
@@ -204,7 +204,7 @@ openat(int         dirfd_,
   LOAD_FUNC(openat);
 
   mode = 0;
-  if(flags_ & O_CREAT)
+  if(flags_ & (O_CREAT|O_TMPFILE))
     {
       va_list args;
       va_start(args,flags_);
@@ -216,7 +216,7 @@ openat(int         dirfd_,
   if(fd == -1)
     return -1;
 
-  if(flags_ & (O_DIRECTORY|O_PATH))
+  if(flags_ & (O_DIRECTORY|O_PATH|O_TMPFILE))
     return fd;
   rv = fstat(fd,&st);
   if(rv == -1)
@@ -253,7 +253,7 @@ openat64(int         dirfd_,
   LOAD_FUNC(openat64);
 
   mode = 0;
-  if(flags_ & O_CREAT)
+  if(flags_ & (O_CREAT|O_TMPFILE))
     {
       va_list args;
       va_start(args,flags_);
@@ -265,7 +265,7 @@ openat64(int         dirfd_,
   if(fd == -1)
     return -1;
 
-  if(flags_ & (O_DIRECTORY|O_PATH))
+  if(flags_ & (O_DIRECTORY|O_PATH|O_TMPFILE))
     return fd;
   rv = fstat(fd,&st);
   if(rv == -1)