Mergerfs creates directories with incorrect permissions when using NFS mounts without no_root_squash

[marktitorenkov]

Describe the bug

I have two machines named truenas and ubuntu running respectively Truenas Scale and Ubunutu Server.
Both machines have a user named user with uid=1000 and gid=1000
Mergerfs is running on the ubuntu machine and truenas exports two NFS shares.
The two shares are exported with the following options on truenas:

user@truenas:~$ cat /etc/exports
"/mnt/tank/files"\
        *(sec=sys,rw,no_subtree_check)
"/mnt/big1/files"\
        *(sec=sys,rw,no_subtree_check)

I have mounted the two shares with the following options on ubuntu:

user@ubuntu:~$ cat /etc/fstab
...
truenas.lan:/mnt/tank/files /mnt/tank/files nfs defaults 0 0
truenas.lan:/mnt/big1/files /mnt/big1/files nfs defaults 0 0
...

I have created a mergerfs pool with the following options on ubuntu

user@ubuntu:~$ cat /etc/fstab
...
/mnt/tank/files:/mnt/big1/files /mnt/merge/files fuse.mergerfs allow_other,cache.files=partial,dropcacheonclose=true,category.action=epall,category.create=epff,category.search=ff,ignorepponrename=true 0 0
...

I have the following files files/directories on both drives:

user@truenas:~$ ls -l /mnt/tank/files/
...
drwxr-xr-x  2 user user   2 Jul 30 03:07 tankdir

user@truenas:~$ ls -l /mnt/big1/files/
...
-rw-r--r-- 1 user user 0 Jul 30 02:53 bigfile

Note that /mnt/big1/files/tankdir/ does not exist.

I attempt to run the following command
user@ubuntu:~$ mv /mnt/merge/files/bigfile /mnt/merge/files/tankdir/

Expected
According to my options category.create=epff,ignorepponrename=true the directory /mnt/big1/files/tankdir/ should be created and /mnt/big1/files/bigfile should be moved inside it.

Actual
The command fails with

mv: cannot move '/mnt/merge/files/bigfile' to '/mnt/merge/files/tankdir/bigfile': Permission denied

The drives look like this afterwards:

user@truenas:~$ ls -l /mnt/tank/files/
...
drwxr-xr-x  2 user user   2 Jul 30 03:07 tankdir

user@truenas:~$ ls -l /mnt/big1/files/
...
-rw-r--r-- 1 user   user    0 Jul 30 02:53 bigfile
drwxr-xr-x 2 nobody nogroup 2 Jul 30 03:07 tankdir

tankdir is created but with the wrong permissions.

Workaround
I change the exports to look like this:

user@truenas:~$ cat /etc/exports
"/mnt/tank/files"\
        *(sec=sys,rw,no_root_squash,no_subtree_check)
"/mnt/big1/files"\
        *(sec=sys,rw,no_root_squash,no_subtree_check)

I run

user@truenas:~$ rmdir /mnt/big1/files/tankdir

I run

user@ubuntu:~$ mv /mnt/merge/files/bigfile /mnt/merge/files/tankdir/

It works as expected.

To Reproduce

1. Create two NFS shares with default options (no_root_squash)
2. Mount them with default options (/mnt/disk1 and /mnt/disk2)
3. Create a directory /mnt/disk1/dir1/
4. Create a file /mnt/disk2/file2
5. Make sure /mnt/disk2/dir1/ does not exist
6. Create a mergerfs mount (/mnt/disk1:/mnt/disk2 /mnt/merge) with the same options as above
7. Attempt mv /mnt/merge/file2 /mnt/merge/dir1/

Actual behavior

• mv fails with operation not permitted
• mergerfs creates /mnt/disk2/dir1/ with owner nobody:nogroup

Expected behavior

• mv succeeds
• mergerfs creates /mnt/disk2/dir1/ with the same owner as /mnt/disk1/dir1/
• /mnt/disk2/file2 is moved into /mnt/disk2/dir1/

System information:

• OS, kernel version: uname -a

Linux ubuntu 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

• mergerfs version: mergerfs -V

mergerfs version: 2.36.0

• List of drives, filesystems, & sizes:
  • df -h

user@ubuntu:~$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
tmpfs                        197M  2.2M  195M   2% /run
/dev/vda2                     63G   21G   39G  35% /
tmpfs                        982M     0  982M   0% /dev/shm
tmpfs                        5.0M     0  5.0M   0% /run/lock
tank/files:big1/files         27T  9.1T   18T  35% /mnt/merge/files
truenas.lan:/mnt/big1/files   17T  2.3T   15T  14% /mnt/big1/files
truenas.lan:/mnt/tank/files   10T  6.9T  3.1T  69% /mnt/tank/files
tmpfs                        197M  4.0K  197M   1% /run/user/1000

• lsblk -f

user@ubuntu:~$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0    7:0    0  63.4M  1 loop /snap/core20/1950
loop1    7:1    0   103M  1 loop /snap/lxd/23541
loop2    7:2    0  63.4M  1 loop /snap/core20/1974
loop3    7:3    0 111.9M  1 loop /snap/lxd/24322
loop4    7:4    0  53.3M  1 loop /snap/snapd/19361
loop5    7:5    0  53.3M  1 loop /snap/snapd/19457
vda    252:0    0    64G  0 disk
├─vda1 252:1    0     1M  0 part
└─vda2 252:2    0    64G  0 part /

• A strace of the application having a problem:
  app.strace.txt
• strace of mergerfs while app tried to do it's thing:
  mergerfs.strace.txt

Additional context

Possible issue
Looking at the trace files it seams that mergerfs first creates the directory as root and then attempts to chown user:user
However when a directory is created as root on NFS without no_root_squash it is mapped to owner nobody:nogroup and afterwards it cannot be modified through NFS. Next the rename operation fails because again even the root user has no write permissions in the newly created directory.

[trapexit]

Without going over all the details you've provided yet... mergerfs expects / requires the ability to run some commands as root that can only be done as root on a standard POSIX filesystem. Root squash naturally breaks that expectation / requirement. No different from some other non-POSIX compliant filesystems.

I can look at adding workarounds for this situation.

[marktitorenkov]

Thanks for the quick response! I agree that having root_squash enabled does introduce some non-standard behaviour. I think that if it is possible to create the necessary directories with the expected permissions immediately instead of chown-ing them later that would be ideal.

[trapexit]

Yeah, it can be done that way but needs to be carefully managed. Can't just change the behavior generally because cloning of paths will not work as non-root in many cases and having to jump to and from root isn't ideal either. Also don't want to perform an action and have to check the result to see if it matches expectation every time as more syscalls means more overhead. root squash behaves in similar ways to some other non-posix filesystems where the expected response of a syscall doesn't match the actual.

Perhaps could allow branches to be marked with their incompatibilities so it can work around it or look to figure it out but at the moment there isn't a easy way to add such options to branches or to act on them due to the lack of a config file. Something I'm working on.

[trapexit]

I'm trying to rewrite the clone path routines to work in this case of squashed root. It requires more syscalls to manage but shouldn't cause any real issues. One thing that did dawn on me though is that NFS acls would be a whole other can of worms (like not being able to delete your own files) so I'm just going to ignore that for now.

[trapexit]

I spent a good amount of time the past few days playing with this and I'm really not sure how well this can work.

If all the perms and ownership line up then it could work in some situations but if paths don't have that they will at best be created with incorrect values. Worse, error out. Remember that it isn't just creating directories. It is trying to clone the path as found on another branch. If foo in foo/bar/baz is not owned by the user making the request or the parent directory doesn't allow creations then I can't do anything.

I suppose I could try anyway but IMO the errors will be confusing and seemingly random to people.