copy_file_range unable to copy file from overlay file system, when file originated from non-overlay file system #2715

ericsciple · 2022-01-04T20:58:43Z

What happened:

Using copy_file_range, copy a file from a non-overlay file system, to the overlay file system
Then attempt to copy the file again, from the overlay file system, to another location on the overlay file system
When attempting the second file copy, copy_file_range cannot copy the file and always copies 0 bytes

What you expected to happen:

copy_file_range is able to copy the file

How to reproduce it (as minimally and precisely as possible):

Follow these three instructions to get a shell to a container running in kubernetes
Copy paste commands from this script

Anything else we need to know?:

Key details:

Reproduces on AKS and EKS (not minikube and not docker)
Uses copy_file_range to perform the copy
Copy a file from non-overlay file system to overlay file system, then from overlay file system to another location on the overlay file system (always copies 0 bytes)
This simple repro demonstrates that non-overlay -> overlay -> overlay fails, but we also see the same issue when attempting non-overlay -> overlay -> non-overlay

Environment:

Kubernetes version (use kubectl version)
- azure cloud shell
Size of cluster (how many worker nodes are in the cluster?)
- 1
General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.)
- nothing, minimal repro
Others:

The text was updated successfully, but these errors were encountered:

ghost · 2022-01-04T20:58:47Z

Hi ericsciple, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
Please abide by the AKS repo Guidelines and Code of Conduct.
If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

ericsciple · 2022-01-04T21:05:58Z

This comment may be related. If so, I wonder whether AKS would need to update to future version of linux to receive the fix?

ghost · 2022-01-07T00:00:42Z

Triage required from @Azure/aks-pm

phealy · 2022-01-07T16:45:43Z

Per this comment the bug was backported into the Ubuntu 5.4 kernel, which is the base for the Ubuntu 5.4.0-azure kernel, which is what AKS nodes use. Once the upstream packages are fixed the patch will get into our next OS update image.

ethomson · 2022-01-26T14:43:08Z

Hi @phealy, I wanted to check in on this. It looks like there's been little to no activity on the Ubuntu bug 1953199. Does this mean that AKS instances are still susceptible to this bug?

ethomson · 2022-02-03T20:39:02Z

Despite that, it looks like this is all resolved in newer images - I'm not able to repro this on Linux 5.4.0-1067-azure #70~18.04.1-Ubuntu SMP. Does that sound right to you @phealy?

ghost · 2022-03-06T01:00:42Z

Action required from @Azure/aks-pm

ghost · 2022-03-21T06:00:44Z