Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docker compose failed to start, csghub-server-1 is unhealthy #894

Open
Tendo33 opened this issue Jan 10, 2025 · 8 comments
Open

docker compose failed to start, csghub-server-1 is unhealthy #894

Tendo33 opened this issue Jan 10, 2025 · 8 comments
Assignees
@Yiling-J
Labels
bug Something isn't working

Comments

@Tendo33
Copy link

Tendo33 commented Jan 10, 2025 •
edited
Loading

Describe the bug

root@fhv-devcsghub-01 /data/nfs/csghub/docker-compose-ori # bash startup.sh
[2025-01-10 15:34:15] [NORM] Current configured domain name is 10.0.36.67.
[2025-01-10 15:34:15] [NORM] Nginx Main:
[2025-01-10 15:34:15] [INFO] - render nginx configuration file.
[2025-01-10 15:34:15] [INFO] - generate temporal auth file.
[2025-01-10 15:34:16] [NORM] CoreDNS:
[2025-01-10 15:34:16] [INFO] - render coredns Corefile.
[2025-01-10 15:34:16] [INFO] - generate CoreDNS reverse parsing files.
[2025-01-10 15:34:16] [NORM] Minio:
[2025-01-10 15:34:16] [INFO] - create data directories.
[2025-01-10 15:34:16] [NORM] Registry:
[2025-01-10 15:34:16] [INFO] - create config directories.
[2025-01-10 15:34:16] [INFO] - generate registry auth file.
[2025-01-10 15:34:16] [NORM] Gitaly:
[2025-01-10 15:34:16] [INFO] - render configuration file.
[2025-01-10 15:34:16] [NORM] Gitlab-Shell:
[2025-01-10 15:34:16] [INFO] - generate gitaly auth file.
[2025-01-10 15:34:16] [INFO] - generate host key pairs for gitlab-shell.
[2025-01-10 15:34:16] [WARN] - ssh_host_rsa_key pair already exists.
[2025-01-10 15:34:16] [WARN] - ssh_host_ecdsa_key pair already exists.
[2025-01-10 15:34:16] [WARN] - ssh_host_ed25519_key pair already exists.
[2025-01-10 15:34:16] [NORM] Csghub_Space_Builder:
[2025-01-10 15:34:16] [INFO] - render docker daemon file.
[2025-01-10 15:34:16] [INFO] - render docker config file.
[2025-01-10 15:34:16] [NORM] Nats:
[2025-01-10 15:34:16] [INFO] - render nats config file.
[2025-01-10 15:34:16] [NORM] Casdoor:
[2025-01-10 15:34:16] [INFO] - render casdoor init_data file.
[2025-01-10 15:34:16] [INFO] - render casdoor config file.
[2025-01-10 15:34:16] [NORM] Csghub_Proxy_Nginx:
[2025-01-10 15:34:16] [INFO] - render proxy nginx config file.
[2025-01-10 15:34:16] [NORM] Starting services...
[+] Running 17/20
 ⠦ Network docker-compose-ori_opencsg                 Created                                                 13.6s 
 ✔ Container docker-compose-ori-nats-1                Started                                                  1.3s 
 ✔ Container docker-compose-ori-redis-1               Started                                                  1.5s 
 ✔ Container docker-compose-ori-registry-1            Started                                                  1.0s 
 ✔ Container docker-compose-ori-postgres-1            Started                                                  1.4s 
 ✔ Container docker-compose-ori-gitaly-1              Started                                                  1.0s 
 ✔ Container docker-compose-ori-minio-1               Started                                                  1.3s 
 ✔ Container docker-compose-ori-temporal-1            Started                                                  2.4s 
 ✔ Container docker-compose-ori-gitlab-shell-1        Started                                                  2.1s 
 ✔ Container docker-compose-ori-casdoor-1             Healthy                                                 13.4s 
 ✔ Container docker-compose-ori-csghub-portal-1       Starte...                                                2.8s 
 ✔ Container docker-compose-ori-temporal-ui-1         Started                                                  3.3s 
 ✘ Container docker-compose-ori-csghub-server-1       Error                                                    7.0s 
 ✔ Container docker-compose-ori-csghub-proxy-1        Started                                                  5.1s 
 ⠸ Container docker-compose-ori-nginx-1               Created                                                 13.3s 
 ✔ Container docker-compose-ori-csghub-mirror-repo-1  S...                                                     4.6s 
 ⠸ Container docker-compose-ori-csghub-db-init-1      Creat...                                                13.3s 
 ✔ Container docker-compose-ori-csghub-user-1         Started                                                  4.6s 
 ✔ Container docker-compose-ori-csghub-mirror-lfs-1   St...                                                    4.8s 
 ✔ Container docker-compose-ori-csghub-accounting-1   St...                                                    5.1s 
dependency failed to start: container docker-compose-ori-csghub-server-1 is unhealthy

root@fhv-devcsghub-01 /data/nfs/csghub/docker-compose-ori # docker logs -f docker-compose-ori-csghub-server-1
Database setup...
Migration init
init logger, level: INFO, format: json
Migration migrate
init logger, level: INFO, format: json
{"time":"2025-01-10T07:34:22.652590318Z","level":"INFO","msg":"there are no new migrations to run (database is up to date)"}
Trigger multisync once in background
Start server...
init logger, level: INFO, format: json
{"time":"2025-01-10T07:34:23.626796384Z","level":"INFO","msg":"FIFOScheduler run started"}
2025/01/10 07:34:23 INFO  No logger configured for temporal client. Created default one.
Error: unable to create workflow client, error: failed reaching server: last connection error: connection error: desc = "transport: Error while dialing: dial tcp 192.171.100.122:7233: connect: connection refused"
Database setup...
Migration init
init logger, level: INFO, format: json
Migration migrate
init logger, level: INFO, format: json
{"time":"2025-01-10T07:34:25.700792256Z","level":"INFO","msg":"there are no new migrations to run (database is up to date)"}
Trigger multisync once in background
Start server...
init logger, level: INFO, format: json
{"time":"2025-01-10T07:34:26.588432132Z","level":"INFO","msg":"FIFOScheduler run started"}
{"time":"2025-01-10T07:34:26.654411105Z","level":"ERROR","msg":"failed to get all service status","error":"Get \"http://csghub-runner:8082/api/v1/service/status-all\": dial tcp: lookup csghub-runner on 127.0.0.11:53: no such host"}
2025/01/10 07:34:26 INFO  No logger configured for temporal client. Created default one.
Error: failed to register cron jobs:  unable to create schedule, error:expected 0 args for function: SyncAsClientWorkflow but found 1
Database setup...
Migration init
init logger, level: INFO, format: json
Migration migrate
init logger, level: INFO, format: json
{"time":"2025-01-10T07:34:30.469571043Z","level":"INFO","msg":"there are no new migrations to run (database is up to date)"}
Trigger multisync once in background
Start server...
init logger, level: INFO, format: json
{"time":"2025-01-10T07:34:30.971572671Z","level":"INFO","msg":"FIFOScheduler run started"}
{"time":"2025-01-10T07:34:31.036007227Z","level":"ERROR","msg":"failed to get all service status","error":"Get \"http://csghub-runner:8082/api/v1/service/status-all\": dial tcp: lookup csghub-runner on 127.0.0.11:53: no such host"}
2025/01/10 07:34:31 INFO  No logger configured for temporal client. Created default one.
Error: failed to register cron jobs:  unable to create schedule, error:expected 0 args for function: SyncAsClientWorkflow but found 1
Database setup...
Migration init
init logger, level: INFO, format: json
Migration migrate
init logger, level: INFO, format: json
{"time":"2025-01-10T07:34:34.094583944Z","level":"INFO","msg":"there are no new migrations to run (database is up to date)"}
Trigger multisync once in background
Start server...
init logger, level: INFO, format: json
{"time":"2025-01-10T07:34:34.500603614Z","level":"INFO","msg":"FIFOScheduler run started"}
2025/01/10 07:34:34 INFO  No logger configured for temporal client. Created default one.
{"time":"2025-01-10T07:34:34.53463916Z","level":"ERROR","msg":"failed to get all service status","error":"Get \"http://csghub-runner:8082/api/v1/service/status-all\": dial tcp: lookup csghub-runner on 127.0.0.11:53: no such host"}
Error: failed to register cron jobs:  unable to create schedule, error:expected 0 args for function: SyncAsClientWorkflow but found 1
Database setup...
Migration init
init logger, level: INFO, format: json
Migration migrate
init logger, level: INFO, format: json

Then I looked at the temporal container's log:

time=2025-01-10T07:34:24.889 level=ERROR msg="failed reaching server: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 192.171.100.122:7233: connect: connection refused\""

Environment
CSGHub Version: v0.7|v0.8|...
OS: Linux | Windows | MacOS..
Hardware: 2c4G | 4c8G |...
Launch: docker compose | helm chart

To Reproduce
Steps to reproduce the behavior:
The latest docker compose file, after changing the host IP in the .env file, executed bash startup.sh.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.
@Tendo33
Copy link
Author

Tendo33 commented Jan 10, 2025 •
edited
Loading

I found that the error seems to be:

{"time":"2025-01-10T08:41:55.275285734Z","level":"ERROR","msg":"failed to get all service status","error":"Get \"http://localhost:8082/api/v1/service/status-all\": dial tcp [::1]:8082: connect: connection refused"}

But i did not run any runner-related functions.

@MasonXon
Copy link
Collaborator

It seems you didn't start by release v1.2.3, It's main branch?

@Tendo33
Copy link
Author

Tendo33 commented Jan 10, 2025

It seems you didn't start by release v1.2.3, It's main branch?

Yes, I am using: https://github.com/OpenCSGs/csghub-installer/tree/main/docker-compose

@wayneliu0019
Copy link
Collaborator

if your csghub is not configured a k8s to work with, then you can ignore below error
{"time":"2025-01-10T08:41:55.275285734Z","level":"ERROR","msg":"failed to get all service status","error":"Get \"http://localhost:8082/api/v1/service/status-all\": dial tcp [::1]:8082: connect: connection refused"} @Tendo33

@MasonXon
Copy link
Collaborator

@Tendo33 Please using package in release v1.2.3, It has a bug in main that we haven't fixed.

@MasonXon
Copy link
Collaborator

MasonXon commented Jan 10, 2025 •
edited
Loading

@wayneliu0019 this error cause csghub-server cannot be started Error: failed to register cron jobs: unable to create schedule, error:expected 0 args for function: SyncAsClientWorkflow but found 1, this is a known bug.

@Rader
Copy link
Collaborator

Rader commented Jan 10, 2025

@Yiling-J pleas have a look

@Rader Rader added the bug Something isn't working label Jan 10, 2025
@Yiling-J
Copy link
Member

@Tendo33 When you use the csghub-installer main branch, it installs the nightly version of CSGHub, which is built every night from the main branch of CSGHub. Please note that the main branch is under active development, and we cannot guarantee stability. To resolve the issue you encountered, please switch to the csghub-installer release branch and try again. sorry for the inconvenience and we’ll work on improving the README to clarify this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Yiling-J Yiling-J

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Rader @Yiling-J @wayneliu0019 @MasonXon @Tendo33