Preheat can't process images from private project correctly

[likunbyl]

Bug report:

I have a private project in the registry, and want to preheat some images:

# curl --request POST 'http://10.2.2.22:8080/api/v1/jobs' --header 'Content-Type: acation/json' --data '{    "type": "preheat",    "args": {        "type": "image",        "url": "https://dockerhub.test.com/v2/base/test/rc/ct/bigdataplatform/confront/cache/manifests/8e915"    }}'
null

At the same time, from manager logs I get the error report:

Error #01: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''

It seems that the preheat api can't handle private projects correctly.

Expected behavior:

Preheat can handle private projects rightly.

How to reproduce it:

Deploy dragonfly 2.0.1 with helm chart, then preheat some images from private projects.

Environment:

• Dragonfly version: 2.0.1
• OS: CentOS Linux 7
• Kernel (e.g. uname -a): 3.10.0-1160.31.1.el7.x86_64
• Docker: docker://19.3.14
• Kubernetes version: v1.19.10

[gaius-qi]

@likunbyl Can you provide a complete log of the manager?

[likunbyl]

[GIN] 2021/10/28 - 10:08:38 | 200 | 98.245µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:08:48 | 200 | 52.819µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:08:58 | 200 | 48.622µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:09:06 | 200 | 43.206µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:09:08 | 200 | 67.241µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:09:18 | 200 | 43.877µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:09:26 | 200 | 66.871µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:09:28 | 200 | 38.305µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:09:38 | 200 | 49.48µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:09:46 | 200 | 39.796µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:09:48 | 200 | 43.892µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:09:58 | 200 | 69.129µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:10:06 | 200 | 41.22µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:10:08 | 200 | 40.231µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:10:18 | 200 | 130.295µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:10:24 | 500 | 73.622869ms | 10.218.2.25 | POST "/api/v1/jobs"
Error #1: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''
[GIN] 2021/10/28 - 10:10:26 | 200 | 49.469µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:10:28 | 200 | 55.24µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:10:38 | 200 | 55.267µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:10:46 | 200 | 108.076µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:10:48 | 200 | 55.463µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:10:58 | 200 | 47.176µs | 10.218.2.25 | GET "/healthy/"
[GIN] 2021/10/28 - 10:11:06 | 200 | 73.44µs | 10.218.2.25 | GET "/healthy/"

[likunbyl]

@gaius-qi these logs are from gin.log, core.log has nothing need attention. Is this enough?

[gaius-qi]

I use distribution.UnmarshalManifest method unmarshal manifest, your problem hits this error. You should take a look at your registry, why doesn’t it meet the OCI Image Manifest Specification.

[likunbyl]

I set the same project to public, the preheat works well. so how can I get the private project preheated? or how can I get the authorize info to the registry?

[jim3ma]

I set the same project to public, the preheat works well. so how can I get the private project preheated? or how can I get the authorize info to the registry?

@gaius-qi Maybe we need support with image credential ?

[gaius-qi]

I set the same project to public, the preheat works well. so how can I get the private project preheated? or how can I get the authorize info to the registry?

@gaius-qi Maybe we need support with image credential ?

Dragonfly 1.0 preheat is done in harbor, so theAuthorization header is passed from harbor. I can add user and password fields to do Basic Authentication with the registry and get permissions.

[likunbyl]

Any update about this issue? @yxxhero

[likunbyl]

Any update about this issue? @jim3ma

[likunbyl]

Any update about this issue? @gaius-qi

[yxxhero]

@likunbyl I will try to see this issue. Thanks very much.

[yxxhero]

1. 引入认证header(auth)，类似于docker的config.json

{
	"auths": {
		"https://index.docker.io/v1/": {
			"auth": "Z29vZ2xlaW1hZ2VzOnl4eDkzMDyxOA=="  username:password base64 encode
		}
	}
}

2. basic auth 获取registry token
   https://github.com/dragonflyoss/Dragonfly2/blob/040815b4b7921dfebcc890666f5aa923e1d9cabd/manager/job/preheat.go#L260
   增加header去请求，即Authorization: Bearer token， 获取授权token

3.拿第二步的token去获取对应的manifest即可

[yxxhero]

https://www.srerun.com/article/2021/4/27/21.html

[gaius-qi]

Use library parsing layer to support manifests in different version, refer to #1204

[likunbyl]

@yxxhero So you mean the support of private project is already in the code logic, could you please update the preheat api document with an example ?

[yxxhero]

@likunbyl I will do this in this weekend.

[1182986209]

i have the same problem. I try to set Authorization header in args, but it doesn't work.

[lilic]

@gaius-qi @yxxhero hello 👋 I noticed that the above preheat supports auth wasn't merged into the main branch, or maybe it was reverted? Is there plans to add this back or does this work differently now? Thank you!

[Panlq]

1. 引入认证header(auth)，类似于docker的config.json

{
	"auths": {
		"https://index.docker.io/v1/": {
			"auth": "Z29vZ2xlaW1hZ2VzOnl4eDkzMDyxOA=="  username:password base64 encode
		}
	}
}

2. basic auth 获取registry token
   https://github.com/dragonflyoss/Dragonfly2/blob/040815b4b7921dfebcc890666f5aa923e1d9cabd/manager/job/preheat.go#L260

   增加header去请求，即Authorization: Bearer token， 获取授权token

3.拿第二步的token去获取对应的manifest即可

这第二步，在最新的 release v2.1.15中仍未修复，https://github.com/dragonflyoss/Dragonfly2/blob/e847992c90a3143f8b02149f2d85f0b0ae019040/manager/job/preheat.go#L267
现有代码对 harbo ，带上 basic token 就可以获取到 manifest ,

https://github.com/dragonflyoss/Dragonfly2/blob/e847992c90a3143f8b02149f2d85f0b0ae019040/manager/job/preheat.go#L182

针对如阿里的私仓，401后(www-authenticate)转发到 auth服务的请求需要带上 basic token 才能获取到正确的 bearer token!!

docker contianerd pull
https://github.com/moby/moby/blob/80a9fc6d36430b067babe841e8332888e24e590b/vendor/github.com/containerd/containerd/remotes/docker/authorizer.go#L240