can not backup data to s3

[saidkind]

here is the error info:

2024-12-17 22:05:00.149 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-17 22:05:00.149 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-17 22:05:00.149 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-17 22:05:00.149 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-17 22:05:00.149 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-17 22:05:00.149 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-17 22:05:00.149 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-17 22:05:00.150 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-17 22:05:00.150 INF pkg/clickhouse/clickhouse.go:327 > clickhouse connection closed
2024-12-17 22:05:00.150 FTL cmd/clickhouse-backup/main.go:668 > error="one of upload table go-routine return error: can't upload: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 1811FBDD150B7034, HostID: 03f6d7ba09b0531a178059659f12e65ab6a75adddf2f548b1f37624d55d95fba, api error IncompleteBody: You did not provide the number of bytes specified by the Content-Length HTTP header."
Good bye!

Here is my config.yml

general:
    remote_storage: s3
    backups_to_keep_local: 7
    backups_to_keep_remote: 31
clickhouse:
    username: default
    password: ""
    host: localhost
    port: 19000
    timeout: 5h
s3:
    access_key: "*****"
    secret_key: "*****"
    bucket: "clickhouse"
    max_parts_count: 400000
    endpoint: "http://localhost:9000"
    part_size: 5242880
    allow_multipart_download: true

there is no problem to execute clickhouse-backup create ,but when I want to execute clickhouse-backup create_remote or clickhouse-backup upload, it can upload a part of the backup file ,but after a few minute later ,the error came out.
It seems like a http problem ,but I couldn't find any config about http

[Slach]

remove

s3:
  max_parts_count: 400000
  part_size: 5242880

these options have good default values

[saidkind]

@Slach thanks for your help, but the same error came out after I remove those two options

[Slach]

could you share full log for the following command?

LOG_LEVEL=debug S3_DEBUG=1 clickhouse-backup upload <your-backup-name>

[Slach]

hmm, interesting
could you share

LOG_LEVEL=debug S3_DEBUG=1 UPLOAD_CONCURRENCY=1 S3_CONCURRENCY=1 clickhouse-backup upload <your-backup-name>

need figure out which request exactly failed

[saidkind]

I have been running this command for about an hour, it runs slowly. I will reply you tomorrow
thanks for your help

[saidkind]

hmm, interesting could you share

LOG_LEVEL=debug S3_DEBUG=1 UPLOAD_CONCURRENCY=1 S3_CONCURRENCY=1 clickhouse-backup upload <your-backup-name>

need figure out which request exactly failed

it succeed , but I don't know why.
I will run this command again and see what will happen
here is the log
backup.log

[Slach]

maybe data in your source table was changed

[saidkind]

it succeed again.

maybe data in your source table was changed

but i just upload the backup files , it's no problem to run 'clickhouse-backup create'.
so when i upload the files, it will check the data once again?

[Slach]

Your original error happened during upload

2024-12-17 22:05:00.150 FTL cmd/clickhouse-backup/main.go:668 > error="one of upload table go-routine return error: can't upload: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 1811FBDD150B7034, HostID: 03f6d7ba09b0531a178059659f12e65ab6a75adddf2f548b1f37624d55d95fba, api error IncompleteBody: You did not provide the number of bytes specified by the Content-Length HTTP header."

so you need to reproduce it and provide full logs to figure out

[Slach]

any news from your side?

[saidkind]

@Slach
i can run this command successfully

LOG_LEVEL=debug S3_DEBUG=1 UPLOAD_CONCURRENCY=1 S3_CONCURRENCY=1 clickhouse-backup upload <your-backup-name>

but i cannot run

clickhouse-backup create_remote

or

clickhouse-backup upload

it happens after i move my database to a new server

now i run the first command to backup my data

if you want to know more information, just tell me want you want me to do

[Slach]

which error message do you see?
provide full logs

[Slach]

any news from your side?