From b0ff2a4b9f8b7ab0e1e0ee996cc8b7702a4262a1 Mon Sep 17 00:00:00 2001 From: Ayush Sethi Date: Wed, 3 May 2023 13:40:48 +0000 Subject: [PATCH] Add about _netdev option in troubleshooting doc (#1062) Add about _netdev option in troubleshooting doc --- docs/troubleshooting.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 9f37943907..a1d63dea7e 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,17 +1,18 @@ # Troubleshooting for production issues This page enumerates some common user facing issues around GCSFuse and also discusses potential solutions to the same. -| Issues | Fix | -|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Generic Mounting Issue| Most of the common mount point issues are around permissions on both local mount point and the Cloud Storage bucket. It is highly recommended to retry with --foreground --debug_fuse --debug_fs --debug_gcs --debug_http flags which would provide much more detailed logs to understand the errors better and possibly provide a solution. | -| Mount successful but files not visible | Try mounting the gcsfuse with --implicit-dir flag. Read the [semantics](https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/semantics.md) to know the reasoning. | -| Mount failed with fusermount3 exit status 1 | It comes when the bucket is already mounted in a folder and we try to mount it again. You need to unmount first and then remount. | -| Mount failed with error: Current requires cgo or $USER set in environment | It comes when we try mounting by building the gcsfuse codebase. To fix this, build the gcsfuse package by enabling the CGO_ENABLED flag in the go env and then mount back.
  1. Check the current value using - ```go env``` command.
  2. If it is unset, set this using - ```export CGO_ENABLED=1``` command.
| -| Mount get stuck with error: DefaultTokenSource: google: could not find default credentials | Run ```gcloud auth application-default login``` command to fetch default credentials to the VM. This will fetch the credentials to the following locations:
  1. For linux - $HOME/.config/gcloud/application_default_credentials.json
  2. For windows - %APPDATA%/gcloud/applicateion_default_credentials.json
| -| Input/Output Error | It’s a generic error, but the most probable culprit is the bucket not having the right permission for Cloud Storage FUSE to operate on. Ref - [here](https://stackoverflow.com/questions/36382704/gcsfuse-input-output-error) | -| Generic NO_PUBKEY Error - while installing Cloud Storage FUSE on ubuntu 22.04 | It happens while running - ```sudo apt-get update``` - working on installing Cloud Storage FUSE. You just have to add the pubkey you get in the error using the below command: ```sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ``` And then try running ```sudo apt-get update``` | -|Cloud Storage FUSE fails with Docker container| Though not tested extensively, the [community](https://stackoverflow.com/questions/65715624/permission-denied-with-gcsfuse-in-unprivileged-ubuntu-based-docker-container) reports that Cloud Storage FUSE works only in privileged mode when used with Docker. There are [solutions](https://cloud.google.com/iam/docs/service-account-overview) which exist and claim to do so without privileged mode, but these are not tested by the Cloud Storage FUSE team | -| daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: fs.NewServer: create file system: SetUpBucket: OpenBucket: Bad credentials for bucket BUCKET_NAME: permission denied | Check the bucket name. Make sure it is within your project. Make sure the applied roles on the bucket contain storage.objects.list permission. You can refer to them [here](https://cloud.google.com/storage/docs/access-control/iam-roles). -| daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: fs.NewServer: create file system: SetUpBucket: OpenBucket: Unknown bucket BUCKET_NAME: no such file or directory | Check the bucket name. Make sure the [service account](https://www.google.com/url?q=https://cloud.google.com/iam/docs/service-accounts&sa=D&source=docs&ust=1679992003850814&usg=AOvVaw3nJ6wNQK4FZdgm8gBTS82l) has permissions to access the files. It must at least have the permissions of the Storage Object Viewer role. | +| Issues | Fix | +|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Generic Mounting Issue | Most of the common mount point issues are around permissions on both local mount point and the Cloud Storage bucket. It is highly recommended to retry with --foreground --debug_fuse --debug_fs --debug_gcs --debug_http flags which would provide much more detailed logs to understand the errors better and possibly provide a solution. | +| Mount successful but files not visible | Try mounting the gcsfuse with --implicit-dir flag. Read the [semantics](https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/semantics.md) to know the reasoning. | +| Mount failed with fusermount3 exit status 1 | It comes when the bucket is already mounted in a folder and we try to mount it again. You need to unmount first and then remount. | +| Mount failed with error: Current requires cgo or $USER set in environment | It comes when we try mounting by building the gcsfuse codebase. To fix this, build the gcsfuse package by enabling the CGO_ENABLED flag in the go env and then mount back.
  1. Check the current value using - ```go env``` command.
  2. If it is unset, set this using - ```export CGO_ENABLED=1``` command.
| +| Mount get stuck with error: DefaultTokenSource: google: could not find default credentials | Run ```gcloud auth application-default login``` command to fetch default credentials to the VM. This will fetch the credentials to the following locations:
  1. For linux - $HOME/.config/gcloud/application_default_credentials.json
  2. For windows - %APPDATA%/gcloud/applicateion_default_credentials.json
| +| Input/Output Error | It’s a generic error, but the most probable culprit is the bucket not having the right permission for Cloud Storage FUSE to operate on. Ref - [here](https://stackoverflow.com/questions/36382704/gcsfuse-input-output-error) | +| Generic NO_PUBKEY Error - while installing Cloud Storage FUSE on ubuntu 22.04 | It happens while running - ```sudo apt-get update``` - working on installing Cloud Storage FUSE. You just have to add the pubkey you get in the error using the below command: ```sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ``` And then try running ```sudo apt-get update``` | +| Cloud Storage FUSE fails with Docker container | Though not tested extensively, the [community](https://stackoverflow.com/questions/65715624/permission-denied-with-gcsfuse-in-unprivileged-ubuntu-based-docker-container) reports that Cloud Storage FUSE works only in privileged mode when used with Docker. There are [solutions](https://cloud.google.com/iam/docs/service-account-overview) which exist and claim to do so without privileged mode, but these are not tested by the Cloud Storage FUSE team | +| daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: fs.NewServer: create file system: SetUpBucket: OpenBucket: Bad credentials for bucket BUCKET_NAME: permission denied | Check the bucket name. Make sure it is within your project. Make sure the applied roles on the bucket contain storage.objects.list permission. You can refer to them [here](https://cloud.google.com/storage/docs/access-control/iam-roles). +| daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: fs.NewServer: create file system: SetUpBucket: OpenBucket: Unknown bucket BUCKET_NAME: no such file or directory | Check the bucket name. Make sure the [service account](https://www.google.com/url?q=https://cloud.google.com/iam/docs/service-accounts&sa=D&source=docs&ust=1679992003850814&usg=AOvVaw3nJ6wNQK4FZdgm8gBTS82l) has permissions to access the files. It must at least have the permissions of the Storage Object Viewer role. | | daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: Mount: mount: running fusermount: exit status 1 stderr: /bin/fusermount: fuse device not found, try 'modprobe fuse' first | To run the container locally, add the --privilege flag to the docker run command: ```docker run --privileged gcr.io/PROJECT/my-fs-app ```
  • You must create a local mount directory
  • If you want all the logs from the mount process use the --foreground flag in combination with the mount command: ```gcsfuse --foreground --debug_gcs --debug_fuse $GCSFUSE_BUCKET $MNT_DIR ```
  • Add --debug_http for HTTP request/response debug output.
  • Add --debug_fuse to enable fuse-related debugging output.
  • Add --debug_gcs to print GCS request and timing information.
-| Cloud Storage FUSE installation fails with an error at build time. | Only specific OS distributions are currently supported, learn more about [Installing Cloud Storage FUSE](https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/installing.md). | +| Cloud Storage FUSE installation fails with an error at build time. | Only specific OS distributions are currently supported, learn more about [Installing Cloud Storage FUSE](https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/installing.md). | +| Cloud Storage FUSE not mounting after reboot when entry is present in ```/etc/fstab``` with 1 or 2 as fsck order | Pass [_netdev option](https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/mounting.md#persisting-a-mount) in fstab entry (reference issue [here](https://github.com/GoogleCloudPlatform/gcsfuse/issues/1043)). With this option, mount will be attempted on reboot only when network is connected.