Scuba on M1

[retu2libc]

Scuba does not seem to work on the M1 macs right now.

Reproduction configuration

image: debian:11

$ scuba ls -a
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/.scuba/scubainit": stat /.scuba/scubainit: no such file or directory: unknown.

Expected Behavior

$ docker run -it --rm -v $(pwd):/test:ro -w /test -u $(id -u):$(id -g) debian:11 ls -a
.  ..

[Kfeavel]

I use scuba with my M1 Pro Mac. The issue is that scuba does not work with Debian based images, but other, non-Debian images work fine. Try using scuba with an alpine image and it should work just fine.

Edit: I'm fairly certain this is related to #109

[retu2libc]

Seems to fail with alpine:latest as well

[Kfeavel]

Can you try with ghcr.io/xyrisos/xyris-build/xyris-build:5.0.0? I've been using that this week and it's worked fine for me.

[retu2libc]

fails with that as well

[Kfeavel]

Oh, interesting. How do you have Docker installed? I'm using Docker Desktop.

[JonathonReinhart]

In general, Suba is not verified to work on Mac. I don't know how Docker works on Mac these days.

I don't think this has anything to do with the image being used. Look at the error message:

stat /.scuba/scubainit: no such file or directory

This makes me suspect that the bind-mounting is not working. That wouldn't surprise me, because that's where we had trouble with OSX in the past. IIRC, only certain host directories could be bind-mounted (presumably due the underlying mechanism used to share directories with the Linux VM) -- and when you specified a directory that wouldn't work, it failed silently, resulting in an empty directory. We bind-mount the .scuba dir, so that looks like a strong possibility of what's happening here. It might depend on how/where you have scuba installed on the host. Try from your home directory or something.

I have neither the bandwidth nor a platform on which to work on this. If anyone wants to contribute, please feel free to volunteer!

[retu2libc]

Oh, interesting. How do you have Docker installed? I'm using Docker Desktop.

Yeah I am as well. iirc i just ran smth like brew install docker --cask

[JonathonReinhart]

If you want to debug this further, I suggest trying scuba --dry-run (docs). This will do two things:

1. Generate the temporary files.
2. Dump out the scuba command line it would run.

You can copy / paste that and play around.

I suspect that if you run it with ls /.scuba, you'll see that scubainit is absent. Other things may or may not be there, depending on whhere they are on the host. You can play around and see what works.

If you figure it out, and can ideally point to documentation about what can(not) be bind-mounted on OSX, I would appreciate a patch to at least detect this problem, and point users at the docs where it is explained and a workaround is provided.

[retu2libc]

yeah the directory exists but its empty. ill look into whats going on a bit.

[retu2libc]

Okay so after some investigation its user error. It escaped me that on macs /var is a symlink to /private/var. After resolving that debian images are stuck on something that appears related to #109 but other images like alpine:latest seem to work.

They fail with

scubainit: Warning: GID 20 already exists in /etc/group
scubainit: Group "staff" already exists with different gid in /etc/group

For anyone who wants a quick and dirty fix just use RUN groupdel staff

[JonathonReinhart]

Okay so after some investigation its user error. It escaped me that on macs /var is a symlink to /private/var. After resolving that

@christheyankee Can you please elaborate? What exactly was the "user error"? How does the fact that /var is a symlink to /private/var cause this issue? How did you resolve it?

Chances are that other Mac users are likely to run into this same problem, so anything we can do now to illuminate the problem saves future hassle for multiple people.

[JonathonReinhart]

The issue is that scuba does not work with Debian based images

@Kfeavel That is a very broad, and concerning statement. debian:10 is used for all of the main tests, so it is definitely not entirely accurate.

If you are aware of a particular failure mode, please open an issue. There may be some specific configuration or subset of images that do not work, but to say "Scuba does not work with Debian based images" without referring to a specific problem is potentially damaging.

[JonathonReinhart]

#109 applies only to images that were built with USER in their Dockerfile. If docker inspect -f '{{ .Config.User }}' $image doesn't output anything, then #109 doesn't apply.

I've moved this debian-images conversation to #197.

[retu2libc]

Okay so after some investigation its user error. It escaped me that on macs /var is a symlink to /private/var. After resolving that

@christheyankee Can you please elaborate? What exactly was the "user error"? How does the fact that /var is a symlink to /private/var cause this issue? How did you resolve it?

Chances are that other Mac users are likely to run into this same problem, so anything we can do now to illuminate the problem saves future hassle for multiple people.

The user error is that you have to allow /private/var/folders in Docker Desktop under Preferences -> Resources -> File sharing. The problem is that when you run with --dry-run you get output that appears to create resources in /var/folder. I then added that to the Docker Desktop resources as described above and the issue persisted. It was only after I realized it was a symlink and then added the real path that it started working.

[retu2libc]

The issue is that scuba does not work with Debian based images

@Kfeavel That is a very broad, and concerning statement. debian:10 is used for all of the main tests, so it is definitely not entirely accurate.

If you are aware of a particular failure mode, please open an issue. There may be some specific configuration or subset of images that do not work, but to say "Scuba does not work with Debian based images" without referring to a specific problem is potentially damaging.

It works fine if you delete the staff group. I've tested this pretty extensively including exposing ports, interacting with daemons, and using some pretty cursed proprietary software. Scuba hasn't died on any of it.

[Kfeavel]

@Kfeavel That is a very broad, and concerning statement. debian:10 is used for all of the main tests, so it is definitely not entirely accurate.

Apologies, I should have clarified that this issue is specific to macOS. I should have been more specific when making my statement.

[JonathonReinhart]

Would either of you care to submit a PR which updates the docs to include battle-tested instructions for getting Scuba working on a Mac?

Specifically:

• What configuration must be set in Docker Desktop? (Scuba on M1 #196 (comment))
• Are there limitations on the directories from which scuba can be run?

I've never owned a Mac and it has been many years since scuba attempted to actually "support" running on a Mac, so I'm both impressed and confused that it works. If you could also clarify this for me, that'd be great:

History

I don't know exactly how Docker on Mac works today, but back in the 2016-2017 timeframe, I believe Docker ("Desktop"?) on OSX ran a boot2docker VM, and then set the DOCKER_HOST environment variable to pipe all local docker commands to that machine. It had some fairly rigid directory-sharing set up so things pretty much worked if you ran Scuba from a project in the right directory. Because of this, a lot of the scubainit stuff I added was unnecessary (the files ended up being owned on the host by your user anyway.) So I added code to basically say "if DOCKER_HOST is set, don't use scubainit: 4beadf2

scuba/src/scuba

Lines 279 to 292 in 4beadf2

	# Determine if Docker is running locally or remotely
	if 'DOCKER_HOST' in os.environ:
	'''
	Docker is running remotely (e.g. boot2docker on OSX).
	We don't need to do any user setup whatsoever.
	TODO: For now, remote instances won't have any .scubainit
	See:
	https://github.com/JonathonReinhart/scuba/issues/17
	'''
	docker_opts = []
	docker_cmd = usercmd
	vol_opts = None

But eventually that one Mac user didn't care anymore, and I had no way of testing it, so I dropped support for it in a refactor leading up to Scuba 2.0: a918d44. Now if DOCKER_HOST was set, Scuba would throw its hands up in the air. That code is still there today:

scuba/scuba/scuba.py

Lines 90 to 100 in b086413

	if self.is_remote_docker:
	"""
	Docker is running remotely (e.g. boot2docker on OSX).
	We don't need to do any user setup whatsoever.
	TODO: For now, remote instances won't have any .scubainit
	See:
	https://github.com/JonathonReinhart/scuba/issues/17
	"""
	raise ScubaError("Remote docker not supported (DOCKER_HOST is set)")

Given that you guys say it works, DOCKER_HOST must not be set, which means:

• Scuba thinks it is operating in "native" mode (see __setup_native_run()). At a minimum this is misleading, but at worst it introduces some inconsistencies vs running truly native on Linux.
• Presumably, the docker daemon is running locally, and connected to via a local UNIX socket? But Docker runs Linux executables, so there is presumably a Linux kernel running somewhere... How does folder sharing into this presumed VM work?

[retu2libc]

How does folder sharing into this presumed VM work?

From a cursory scan of docker.com it looks like a FUSE client and server connected via Hypervisor sockets.

Given that you guys say it works, DOCKER_HOST must not be set

That would line up with what I'm seeing.

$ echo $DOCKER_HOST

$

Would either of you care to submit a PR which updates the docs to include battle-tested instructions for getting Scuba working on a Mac?

I'd be happy to do this but I think it might be better to hold off until we decide what to do about #197 since Debian is such a common choice for docker images.