We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
当dfget因为别的原因拉取HDFS上的文件失败后, 在 https://github.com/dragonflyoss/Dragonfly2/blob/v2.0.1/client/dfget/dfget.go#L156 触发回源并作流的写入操作 而io.Copy最终会调用一次 https://github.com/dragonflyoss/Dragonfly2/blob/v2.0.1/pkg/source/hdfsprotocol/hdfs_source_client.go#L286 处的函数, 但该函数只作了一次最大为buf size的写入操作就返回结果,导致最终文件仅有512字节,并且程序误判为成功。 将函数改为
func (rc *hdfsFileReaderClose) WriteTo(w io.Writer) (n int64, err error) { return io.Copy(w, rc.limited) }
后成功回源下载文件
最后补充一下,回源前下载失败是由于
storage: # 磁盘 GC 阈值,缓存数据超过阈值后,最旧的缓存数据将会被清理 diskGCThreshold: 50Gi
该配置写的太小引起,可以考虑一下GC掉正在进行的任务的合理性
在GC阈值分别设为10G、50G时下载HDFS上60G的文件都能复现
uname -a
The text was updated successfully, but these errors were encountered:
感谢反馈。
代码修改,能否提个 PR,我们来负责 review ?
Sorry, something went wrong.
Successfully merging a pull request may close this issue.
Bug report:
当dfget因为别的原因拉取HDFS上的文件失败后,
在 https://github.com/dragonflyoss/Dragonfly2/blob/v2.0.1/client/dfget/dfget.go#L156 触发回源并作流的写入操作
而io.Copy最终会调用一次 https://github.com/dragonflyoss/Dragonfly2/blob/v2.0.1/pkg/source/hdfsprotocol/hdfs_source_client.go#L286 处的函数,
但该函数只作了一次最大为buf size的写入操作就返回结果,导致最终文件仅有512字节,并且程序误判为成功。
将函数改为
后成功回源下载文件
最后补充一下,回源前下载失败是由于
该配置写的太小引起,可以考虑一下GC掉正在进行的任务的合理性
Expected behavior:
How to reproduce it:
在GC阈值分别设为10G、50G时下载HDFS上60G的文件都能复现
Environment:
uname -a
):4.9.0The text was updated successfully, but these errors were encountered: