opsira

Mounting keys and certificates into containers

In short

The file is there, the path is right, and the application cannot read it. Almost always this is which user owns it, not what mode it has.

The mismatch

Bind mounts pass through numeric user and group ids, not names. A file owned by root on the host is owned by uid 0 inside the container. If the process runs as a different uid, and the file is not world readable, it cannot open it.

The confusing part is that everything looks correct. The file exists at the expected path, the mount is right, and the mode looks sensible. The error is a permission denied that seems to contradict what you can see.

Diagnose from inside

docker compose exec service id
docker compose exec service ls -ln /path/to/secret

Use the numeric listing. Names are resolved differently inside the container and will mislead you. Compare the uid the process runs as against the uid that owns the file, and the answer is usually immediate.

The fix

Change ownership on the host to the uid the container process uses, and keep the mode restrictive:

chown -R 1000:1000 /path/to/certs
chmod 600 /path/to/certs/*.key

Do not solve it by making the file world readable. That works, and it puts a private key where every process on the host can read it. Do not solve it by running the container as root either, for the same class of reason.

Directories need executing

A readable file inside a directory the user cannot traverse is still unreachable. Directories need the execute bit for the owner, which is easy to lose when copying permissions from a file.

Keep private keys off shared paths

Mount the specific directory a service needs, read only where possible, rather than a broad parent. If a key only ever needs to be read by one service, it should not be visible to the others.

And keep the key out of the image itself. A secret copied in at build time is in every layer, every registry copy and every backup of that image, long after you have rotated it.

Need help with any of this?

These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.