folders not mounting on reboot

[gcontois]

I recently noticed that upon reboot my gcsfuse folders set up in my fstab file are not getting mounted. This is a new occurrence even after the recent permission issues. If i manually run sudo mount -a, everything is fine.

fstab file:
ctwebcust-live-prwdr /var/www/html/prwdr gcsfuse rw,allow_other,file_mode=777,dir_mode=777,implicit_dirs,key_file=/usr/key.json
mspeechsample-prod /var/www/html/mspeechsample gcsfuse rw,allow_other,file_mode=777,dir_mode=777,implicit_dirs,key_file=/usr/key.json
restapiresources-prod /var/www/html/restapiresources gcsfuse rw,allow_other,file_mode=777,dir_mode=777,implicit_dirs,key_file=/usr/key.json
prwrw-prod /var/www/html/prwrw gcsfuse rw,allow_other,file_mode=777,dir_mode=777,implicit_dirs,key_file=/usr/key.json

Log after reboot: (they do not get mounted)
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[445]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /var/www/html/restapiresources\n","timestampSeconds":1680356210,"timestampNanos":965838198}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[445]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Opening GCS connection...\n","timestampSeconds":1680356210,"timestampNanos":965940118}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[432]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /var/www/html/prwrw\n","timestampSeconds":1680356210,"timestampNanos":967719040}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[432]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Opening GCS connection...\n","timestampSeconds":1680356210,"timestampNanos":967794600}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[433]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /var/www/html/prwdr\n","timestampSeconds":1680356210,"timestampNanos":971932295}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[433]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Opening GCS connection...\n","timestampSeconds":1680356210,"timestampNanos":972039156}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[450]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /var/www/html/mspeechsample\n","timestampSeconds":1680356210,"timestampNanos":973770968}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[450]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Opening GCS connection...\n","timestampSeconds":1680356210,"timestampNanos":973838078}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[445]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a mount at "/var/www/html/restapiresources"\n","timestampSeconds":1680356211,"timestampNanos":91893470}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[450]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a mount at "/var/www/html/mspeechsample"\n","timestampSeconds":1680356211,"timestampNanos":91908330}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[450]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a new server...\n","timestampSeconds":1680356211,"timestampNanos":91938420}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[445]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a new server...\n","timestampSeconds":1680356211,"timestampNanos":91938440}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[445]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Set up root directory for bucket restapiresources-prod\n","timestampSeconds":1680356211,"timestampNanos":91950260}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[450]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Set up root directory for bucket mspeechsample-prod\n","timestampSeconds":1680356211,"timestampNanos":91950250}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[432]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a mount at "/var/www/html/prwrw"\n","timestampSeconds":1680356211,"timestampNanos":92131270}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[432]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a new server...\n","timestampSeconds":1680356211,"timestampNanos":92159230}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[433]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a mount at "/var/www/html/prwdr"\n","timestampSeconds":1680356211,"timestampNanos":92150700}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[432]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Set up root directory for bucket prwrw-prod\n","timestampSeconds":1680356211,"timestampNanos":92168510}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[433]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a new server...\n","timestampSeconds":1680356211,"timestampNanos":92168670}
Apr 1 13:38:31 ctwebfuse-hc-l gcsfuse[433]: {"name":"root","levelname":"INFO","severity":"INFO","message":"Set up root directory for bucket ctwebcust-live-prwdr\n","timestampSeconds":1680356211,"timestampNanos":92207370}

Returned after a manual sudo mount -a
Calling gcsfuse with arguments: --implicit-dirs=true --key-file /usr/key.json -o rw -o allow_other --file-mode 777 --dir-mode 777 ctwebcust-live-prwdr /var/www/html/prwdr
2023/04/01 13:42:26.429929 Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /var/www/html/prwdr
Calling gcsfuse with arguments: --implicit-dirs=true --key-file /usr/key.json -o rw -o allow_other --file-mode 777 --dir-mode 777 mspeechsample-prod /var/www/html/mspeechsample
2023/04/01 13:42:26.671311 Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /var/www/html/mspeechsample
Calling gcsfuse with arguments: -o rw -o allow_other --file-mode 777 --dir-mode 777 --implicit-dirs=true --key-file /usr/key.json restapiresources-prod /var/www/html/restapiresources
2023/04/01 13:42:26.813454 Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /var/www/html/restapiresources
Calling gcsfuse with arguments: -o rw -o allow_other --file-mode 777 --dir-mode 777 --implicit-dirs=true --key-file /usr/key.json prwrw-prod /var/www/html/prwrw
2023/04/01 13:42:26.931168 Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /var/www/html/prwrw

Just wondering if I am missing something. As stated this was working up to recently.

[raj-prince]

@gcontois, Sorry for the late response!

Can you please provide the logs after adding more debug flags while mounting gcsfuse - --debug_gcs, --debug_fs, --debug_fuse? Please don't forget to add --log-file flag, to collect all the logs.

Regards,
Prince Kumar.

[gcontois]

I set this up on a non live server.
fstab file:
LABEL=cloudimg-rootfs / ext4 defaults 0 1
LABEL=UEFI /boot/efi vfat umask=0077 0 1

ctwebcust-test-prwdr /var/www/html/prwdr gcsfuse rw,allow_other,file_mode=777,dir_mode=777,implicit_dirs,debug_gcs,debug_fs,debug_fuse,log_file=/var/log/fuse.log,key_file=/usr/key.json

Log after reboot: (does not get mounted) - just seems to stop
{"name":"root","levelname":"INFO","severity":"INFO","message":"Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /var/www/html/prwdr\n","timestampSeconds":1681132871,"timestampNanos":809785695}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Opening GCS connection...\n","timestampSeconds":1681132871,"timestampNanos":809927336}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a mount at "/var/www/html/prwdr"\n","timestampSeconds":1681132871,"timestampNanos":869427943}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a new server...\n","timestampSeconds":1681132871,"timestampNanos":869474692}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Set up root directory for bucket ctwebcust-test-prwdr\n","timestampSeconds":1681132871,"timestampNanos":869503712}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req 0x0: \u003c- ListObjects("")\n","timestampSeconds":1681132871,"timestampNanos":869520903}

Returned after a manual sudo mount -a
{"name":"root","levelname":"INFO","severity":"INFO","message":"Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /var/www/html/prwdr\n","timestampSeconds":1681132871,"timestampNanos":809785695}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Opening GCS connection...\n","timestampSeconds":1681132871,"timestampNanos":809927336}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a mount at "/var/www/html/prwdr"\n","timestampSeconds":1681132871,"timestampNanos":869427943}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a new server...\n","timestampSeconds":1681132871,"timestampNanos":869474692}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Set up root directory for bucket ctwebcust-test-prwdr\n","timestampSeconds":1681132871,"timestampNanos":869503712}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req 0x0: \u003c- ListObjects("")\n","timestampSeconds":1681132871,"timestampNanos":869520903}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Start gcsfuse/0.42.3 (Go version go1.19.5) for app "" using mount point: /var/www/html/prwdr\n","timestampSeconds":1681133139,"timestampNanos":58349370}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Opening GCS connection...\n","timestampSeconds":1681133139,"timestampNanos":58632200}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a mount at "/var/www/html/prwdr"\n","timestampSeconds":1681133139,"timestampNanos":118089066}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Creating a new server...\n","timestampSeconds":1681133139,"timestampNanos":118186606}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Set up root directory for bucket ctwebcust-test-prwdr\n","timestampSeconds":1681133139,"timestampNanos":121399457}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req 0x0: \u003c- ListObjects("")\n","timestampSeconds":1681133139,"timestampNanos":121569027}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req 0x0: -\u003e ListObjects("") (139.9024ms): OK\n","timestampSeconds":1681133139,"timestampNanos":261479107}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Mounting file system "ctwebcust-test-prwdr"...\n","timestampSeconds":1681133139,"timestampNanos":261928837}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Beginning the mounting kickoff process\n","timestampSeconds":1681133139,"timestampNanos":262078687}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Parsing fuse file descriptor\n","timestampSeconds":1681133139,"timestampNanos":262092637}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Preparing for direct mounting\n","timestampSeconds":1681133139,"timestampNanos":262134047}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Successfully opened the /dev/fuse in blocking mode\n","timestampSeconds":1681133139,"timestampNanos":262167557}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Starting the unix mounting\n","timestampSeconds":1681133139,"timestampNanos":262204567}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Unix mounting completed successfully\n","timestampSeconds":1681133139,"timestampNanos":265095568}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Completed the mounting kickoff process\n","timestampSeconds":1681133139,"timestampNanos":265113558}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Creating a connection object\n","timestampSeconds":1681133139,"timestampNanos":265121418}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00000002 connection.go:416] \u003c- init\n","timestampSeconds":1681133139,"timestampNanos":277011133}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00000002 connection.go:498] -\u003e OK ()\n","timestampSeconds":1681133139,"timestampNanos":277085693}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Successfully created the connection\n","timestampSeconds":1681133139,"timestampNanos":277159034}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Waiting for mounting process to complete\n","timestampSeconds":1681133139,"timestampNanos":277173954}
{"name":"root","levelname":"INFO","severity":"INFO","message":"File system has been successfully mounted.\n","timestampSeconds":1681133139,"timestampNanos":277180583}

THANKS

[sethiay]

Hey @gcontois , Thanks for sharing the logs.

In the logs that you have shared, the ListObjects method is stuck before running sudo mount -a. We aren't able to reproduce this exactly. However, we suspect that this is happening due to network connection not available just after reboot at the time of mounting. (Network connection is required at the time of mounting because gcsfuse tries to list the bucket before mounting as a check)

Could you please try adding _netdev option in /etc/fstab entry for gcsfuse and confirm if you still face the same issue again ? So, the fstab entry should like:
ctwebcust-test-prwdr /var/www/html/prwdr gcsfuse rw,allow_other,_netdev,file_mode=777,dir_mode=777,implicit_dirs,debug_gcs,debug_fs,debug_fuse,log_file=/var/log/fuse.log,key_file=/usr/key.json

[gcontois]

This seems to have fix the issue. I added _netdev to all my connections.
Thank you very much.

[0355238421]

0973026218