RHEL9 podman 'docker' cannot write to mounted volumes

[dslarm]

On a RHEL9/Rocky 9 instance, the default 'docker' is podman - which wraps the docker commands, and runs in userspace instead of via a docker daemon running as root.

With bioconda-utils build, the mounted volumes such as "-v /home/rocky/miniforge3/envs/bioconda/conda-bld:/opt/host-conda-bld" are mounted as root owned. This prevents 'conda' user from writing to them - and errors almost immediately as it tries to create (as conda) a linux-64 subdirectory at that mount point.

On the same system, conda's own image and "build-locally.py" works.

The docker command under the hood is:

docker run -t --net host --rm -v /tmp/tmpi0zk8bk8/build_script.bash:/opt/build_script.bash:rw,z,delegated -v /home/rocky/miniforge3/envs/bioconda/conda-bld:/opt/host-conda-bld:rw,z,delegated -v /home/rocky/bioconda-recipes/recipes/bwa:/opt/recipe:rw,z,delegated -e LANG=C.UTF-8 -e HOST_USER_ID=1000 quay.io/bioconda/bioconda-utils-build-env-cos7:3.7.0 /bin/bash /opt/build_script.bash

I tried a few things - so to save someone else some time, here's what I did:

• There are some differences in Conda's build-locally.py - it adds ":rw,z,delegated" as options to the volume mount ('-v source:dest:rw,z,delegated'). I tried that locally with bioconda-utils - it did not work.
• I also ran locally as root executing the container launch - doesn't fix it.
• I ran with "-u root" to the podman command - doesn't fix it.
• I tried ":U" as an option to the mount (the "2024 way to do this" according to some) - doesn't fix it.
• I tried --userns=keep-id - doesn't fix it
• I found that the no matter what I did, the whole of /opt/build_script.bash executed as 'conda' - which means the chown at the end of that script couldn't fix the mounted directory's problem.
• Just running bash inside the container (run -it ... bash) also dropped me into a shell there with conda user: instead of as the container's root - which I couldn't fix with any incantation of "-u" or even an outside (ie. host side) sudo.

Finally: the workaround that works - I uninstalled the podman completely, and instead - from the docker website's instructions (https://docs.docker.com/engine/install/rhel/) - installed the real docker - this time things work.

That's a workaround - podman isn't going away so this issue is going to bite again. I don't have familiarity with the container infrastructure to suggest a fix. Either there is an incantation I have not tried to podman, or there is an issue with the prebuilt container itself.