Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

指定md5校验下载失败 #1199

Closed
wciq1208 opened this issue Mar 25, 2022 · 2 comments
Closed

指定md5校验下载失败 #1199

wciq1208 opened this issue Mar 25, 2022 · 2 comments
Assignees

Comments

@wciq1208
Copy link

wciq1208 commented Mar 25, 2022

Bug report:

使用--digest参数指定摘要校验结果时下载失败,产生如下错误日志,可以看出是校验时没将用户输入的摘要类型前缀去除所导致

···
{"level":"warn","ts":"2022-03-25 18:29:25.626","caller":"digestutils/digest_reader.go:74","msg":"digest not match, desired: md5:965a9b47df62167a2a58e08463f965a6, actual: 965a9b47df62167a2a58e08463f965a6","peer":"10.144.237.247-658-0cad55ec-914f-49cb-b374-e9706d52d9d7-default","task":"dc1dafe7a2c51190715306d9527678a94a40dad221e788b1703ef699fc8c2b47","component":"PeerTask","stacktrace":"d7y.io/dragonfly/v2/pkg/util/digestutils.(*digestReader).Read\n\t/mnt/sdb1/Dragonfly2/pkg/util/digestutils/digest_reader.go:74\nd7y.io/dragonfly/v2/pkg/util/digestutils.(*digestReader).Read\n\t/mnt/sdb1/Dragonfly2/pkg/util/digestutils/digest_reader.go:64\nio.(*LimitedReader).Read\n\t/mnt/sdb1/tools/go/src/io/io.go:473\nio.copyBuffer\n\t/mnt/sdb1/tools/go/src/io/io.go:423\nio.Copy\n\t/mnt/sdb1/tools/go/src/io/io.go:382\nos.genericReadFrom\n\t/mnt/sdb1/tools/go/src/os/file.go:162\nos.(*File).ReadFrom\n\t/mnt/sdb1/tools/go/src/os/file.go:156\nio.copyBuffer\n\t/mnt/sdb1/tools/go/src/io/io.go:409\nio.Copy\n\t/mnt/sdb1/tools/go/src/io/io.go:382\nd7y.io/dragonfly/v2/client/daemon/storage.(*localTaskStore).WritePiece\n\t/mnt/sdb1/Dragonfly2/client/daemon/storage/local_storage.go:125\nd7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).processPieceFromSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/piece_manager.go:206\nd7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).downloadUnknownLengthSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/piece_manager.go:378\nd7y.io/dragonfly/v2/client/daemon/peer.(*pieceManager).DownloadSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/piece_manager.go:297\nd7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).backSource\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/peertask_conductor.go:387\nd7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).waitFirstPeerPacket\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/peertask_conductor.go:823\nd7y.io/dragonfly/v2/client/daemon/peer.(*peerTaskConductor).pullPiecesFromPeers\n\t/mnt/sdb1/Dragonfly2/client/daemon/peer/peertask_conductor.go:692"}
···

Expected behavior:


https://github.com/dragonflyoss/Dragonfly2/blob/v2.0.2-rc.25/pkg/util/digestutils/digest_reader.go#L63
该处Read函数改为下述代码后不再出错

func (dr *digestReader) Read(p []byte) (int, error) {
	n, err := dr.r.Read(p)
	if err != nil && err != io.EOF {
		return n, err
	}
	if n > 0 {
		dr.hash.Write(p[:n])
	}
	if err == io.EOF && dr.digest != "" {
		digest := dr.Digest()
		inputDigest := dr.digest
		if strings.Contains(inputDigest, ":") {
			digestInfo := strings.SplitN(inputDigest, ":", 2)
			inputDigest = digestInfo[1]
		}
		if digest != inputDigest {
			dr.Warnf("digest not match, desired: %s, actual: %s", inputDigest, digest)
			return n, ErrDigestNotMatch
		}
		dr.Debugf("digest match: %s", digest)
	}
	return n, err
}

How to reproduce it:

/dragonfly/bin/dfget --timeout 1027s --url hdfs://jssz-bigdata-cluster//department/ai/recsys/athena_data_export/shjd/mxnet2tf_v32/2c4b1fb1af9726ad05749fb348a8d4d6-1648224945-71470831/000000/slot-0.0.0.0-0-1648185845-47-47435/wdctr.data-00000-of-00001 --console --show-progress -O /dragonfly_cache/wdctr.data-00000-of-00001 -n --verbose --digest md5:290667593c3c39e71d99eaf9aa68434b --pattern p2p --config /etc/dragonfly/dfget.yaml

Environment:

  • Dragonfly version:v2.0.2-rc.25
  • OS:debian 10
  • Kernel (e.g. uname -a):4.9.0-8-amd64 [WIP] Implement df daemon #1 SMP Debian 4.9.144-3 (2019-02-02) x86_64 GNU/Linux
  • Others:
@gaius-qi
Copy link
Member

@244372610

@gaius-qi
Copy link
Member

Please try latest version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants