We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
使用--digest参数指定摘要校验结果时下载失败,产生如下错误日志,可以看出是校验时没将用户输入的摘要类型前缀去除所导致
··· {"level":"warn","ts":"2022-03-25 18:29:25.626","caller":"digestutils/digest_reader.go:74","msg":"digest not match, desired: md5:965a9b47df62167a2a58e08463f965a6, actual: 965a9b47df62167a2a58e08463f965a6","peer":"10.144.237.247-658-0cad55ec-914f-49cb-b374-e9706d52d9d7-default","task":"dc1dafe7a2c51190715306d9527678a94a40dad221e788b1703ef699fc8c2b47","component":"PeerTask","stacktrace":"d7y.io/dragonfly/v2/pkg/util/digestutils.(*digestReader).Read\n\t/mnt/sdb1/Dragonfly2/pkg/util/digestutils/digest_reader.go:74\nd7y.io/dragonfly/v2/pkg/util/digestutils.(*digestReader).Read\n\t/mnt/sdb1/Dragonfly2/pkg/util/digestutils/digest_reader.go:64\nio.(*LimitedReader).Read\n\t/mnt/sdb1/tools/go/src/io/io.go:473\nio.copyBuffer\n\t/mnt/sdb1/tools/go/src/io/io.go:423\nio.Copy\n\t/mnt/sdb1/tools/go/src/io/io.go:382\nos.genericReadFrom\n\t/mnt/sdb1/tools/go/src/os/file.go:162\nos.(*File).ReadFrom\n\t/mnt/sdb1/tools/go/src/os/file.go:156\nio.copyBuffer\n\t/mnt/sdb1/tools/go/src/io/io.go:409\nio.Copy\n\t/mnt/sdb1/tools/go/src/io/io.go:382\nd7y.io/dragonfly/v2/client/daemon/storage.(*localTaskStore).WritePiece\n\t/mnt/sdb1/Dragonfly2/client/daemon/storage/local_storage.go:125\nd7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).processPieceFromSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/piece_manager.go:206\nd7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).downloadUnknownLengthSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/piece_manager.go:378\nd7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).DownloadSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/piece_manager.go:297\nd7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).backSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/peertask_conductor.go:387\nd7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).waitFirstPeerPacket\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/peertask_conductor.go:823\nd7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPiecesFromPeers\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/peertask_conductor.go:692"} ···
将 https://github.com/dragonflyoss/Dragonfly2/blob/v2.0.2-rc.25/pkg/util/digestutils/digest_reader.go#L63 该处Read函数改为下述代码后不再出错
func (dr *digestReader) Read(p []byte) (int, error) { n, err := dr.r.Read(p) if err != nil && err != io.EOF { return n, err } if n > 0 { dr.hash.Write(p[:n]) } if err == io.EOF && dr.digest != "" { digest := dr.Digest() inputDigest := dr.digest if strings.Contains(inputDigest, ":") { digestInfo := strings.SplitN(inputDigest, ":", 2) inputDigest = digestInfo[1] } if digest != inputDigest { dr.Warnf("digest not match, desired: %s, actual: %s", inputDigest, digest) return n, ErrDigestNotMatch } dr.Debugf("digest match: %s", digest) } return n, err }
/dragonfly/bin/dfget --timeout 1027s --url hdfs://jssz-bigdata-cluster//department/ai/recsys/athena_data_export/shjd/mxnet2tf_v32/2c4b1fb1af9726ad05749fb348a8d4d6-1648224945-71470831/000000/slot-0.0.0.0-0-1648185845-47-47435/wdctr.data-00000-of-00001 --console --show-progress -O /dragonfly_cache/wdctr.data-00000-of-00001 -n --verbose --digest md5:290667593c3c39e71d99eaf9aa68434b --pattern p2p --config /etc/dragonfly/dfget.yaml
uname -a
The text was updated successfully, but these errors were encountered:
@244372610
Sorry, something went wrong.
Please try latest version.
244372610
Successfully merging a pull request may close this issue.
Bug report:
使用--digest参数指定摘要校验结果时下载失败,产生如下错误日志,可以看出是校验时没将用户输入的摘要类型前缀去除所导致
···
{"level":"warn","ts":"2022-03-25 18:29:25.626","caller":"digestutils/digest_reader.go:74","msg":"digest not match, desired: md5:965a9b47df62167a2a58e08463f965a6, actual: 965a9b47df62167a2a58e08463f965a6","peer":"10.144.237.247-658-0cad55ec-914f-49cb-b374-e9706d52d9d7-default","task":"dc1dafe7a2c51190715306d9527678a94a40dad221e788b1703ef699fc8c2b47","component":"PeerTask","stacktrace":"d7y.io/dragonfly/v2/pkg/util/digestutils.(*digestReader).Read\n\t/mnt/sdb1/Dragonfly2/pkg/util/digestutils/digest_reader.go:74\nd7y.io/dragonfly/v2/pkg/util/digestutils.(*digestReader).Read\n\t/mnt/sdb1/Dragonfly2/pkg/util/digestutils/digest_reader.go:64\nio.(*LimitedReader).Read\n\t/mnt/sdb1/tools/go/src/io/io.go:473\nio.copyBuffer\n\t/mnt/sdb1/tools/go/src/io/io.go:423\nio.Copy\n\t/mnt/sdb1/tools/go/src/io/io.go:382\nos.genericReadFrom\n\t/mnt/sdb1/tools/go/src/os/file.go:162\nos.(*File).ReadFrom\n\t/mnt/sdb1/tools/go/src/os/file.go:156\nio.copyBuffer\n\t/mnt/sdb1/tools/go/src/io/io.go:409\nio.Copy\n\t/mnt/sdb1/tools/go/src/io/io.go:382\nd7y.io/dragonfly/v2/client/daemon/storage.(*localTaskStore).WritePiece\n\t/mnt/sdb1/Dragonfly2/client/daemon/storage/local_storage.go:125\nd7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).processPieceFromSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/piece_manager.go:206\nd7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).downloadUnknownLengthSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/piece_manager.go:378\nd7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).DownloadSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/piece_manager.go:297\nd7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).backSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/peertask_conductor.go:387\nd7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).waitFirstPeerPacket\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/peertask_conductor.go:823\nd7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPiecesFromPeers\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/peertask_conductor.go:692"}
···
Expected behavior:
将
https://github.com/dragonflyoss/Dragonfly2/blob/v2.0.2-rc.25/pkg/util/digestutils/digest_reader.go#L63
该处Read函数改为下述代码后不再出错
How to reproduce it:
Environment:
uname -a
):4.9.0-8-amd64 [WIP] Implement df daemon #1 SMP Debian 4.9.144-3 (2019-02-02) x86_64 GNU/LinuxThe text was updated successfully, but these errors were encountered: