Use mirror server instead of GCS bucket for SDK tar balls and binary packages

[pothos]

Description

Both cork's way of downloading the SDK tar ball and the portage binary package URLs in the SDK and the developer container make use of the GCS bucket. However, we will immediately move this content to the regular release server that hosts the other release assets.

Impact

Users can't set up the SDK or use binary packages in the developer container to compile kernel modules or similar, until we have published the content at the new location and changed the portage URLs for new releases and document how to set them for old releases, and published a new cork release that uses the new location.

[jonasclaes]

What's the current status? How long do you estimate the move? I would love to develop something for Hacktober, but I can't because of this issue at the moment.

[pothos]

You can find the SDKs now on https://mirror.release.flatcar-linux.net/sdk/ (tar balls and binary packages for /etc/portage/make.conf), and the board packages on https://mirror.release.flatcar-linux.net/boards/ for /build/BOARD/etc/portage/make.conf in the SDK board or for /etc/portage/make.conf in the developer container.
The SDK download via cork needs a source code change and is not yet done.

[miao0miao]

@jonasclaes we are currently working on a new release to all channels that will fix this issue, it should be ready by the end of this week.

[tormath1]

Linking related PRs:

• cork: add --json-key to create and download mantle#239: provide a --json-key argument to pull artifacts from private GCS buckets
• sdk: Updated SDK URL mantle#238 (cork: add --sdk-url mantle#240) provide a --sdk-url to modify the SDK location (default to mirror.release...)
• jenkins: add --json-key and --sdk-url to cork update commands scripts#158: include the --json-key changes to the CI scripts
• setup_board, update_chroot, dev container: use new bincache mirror scripts#159

[pothos]

Downloading the SDK works again with the latest cork: https://github.com/flatcar-linux/mantle/releases/tag/v0.17.0

[tbbharaj]

Hello, I tried with latest cork and getting the same download failure error with https://github.com/flatcar-linux/mantle/releases/tag/v0.17.0

2021-10-06T21:42:28Z cli: Started logging at level INFO
2021-10-06T21:42:28Z cork: Detecting SDK version
2021-10-06T21:42:29Z cork: Found SDK version 2801.0.0 from remote repo
2021-10-06T21:42:29Z cork: Downloading SDK version 2801.0.0
2021-10-06T21:42:29Z sdk: Downloading https://mirror.release.flatcar-linux.net/sdk/amd64/2801.0.0/flatcar-sdk-amd64-2801.0.0.tar.bz2 to /home/ubuntu/flatcar-sdk/.cache/sdks/flatcar-sdk-amd64-2801.0.0.tar.bz2
2021-10-06T21:42:34Z cork: Download failed: 404 Not Found: https://mirror.release.flatcar-linux.net/sdk/amd64/2801.0.0/flatcar-sdk-amd64-2801.0.0.tar.bz2

[jepio]

Hi @tbbharaj, we prioritized copying current releases over (check https://mirror.release.flatcar-linux.net/sdk/amd64/). Are you able to update to a newer version?

[tbbharaj]

@jepio Thanks I was able to pull latest version. I am trying to build image from source following steps here: https://kinvolk.io/docs/flatcar-container-linux/latest/reference/developer-guides/sdk-modifying-flatcar/

1. Using latest cork 0.17.0
2. Downloaded latest version 3005.0.0

But still when I do ./setup_board --board=amd64-usr I am getting following error:

!!! Error fetching binhost package info from 'https://storage.googleapis.com/flatcar-jenkins/boards/amd64-usr/2905.0.0/pkgs'
!!! HTTP Error 403: Forbidden

!!! Error fetching binhost package info from 'https://storage.googleapis.com/flatcar-jenkins/boards/amd64-usr/2905.0.0/toolchain'
!!! HTTP Error 403: Forbidden

!!! Error fetching binhost package info from 'https://storage.googleapis.com/flatcar-jenkins/sdk/amd64/2905.0.0/toolchain'
!!! HTTP Error 403: Forbidden

!!! Error fetching binhost package info from 'https://storage.googleapis.com/flatcar-jenkins/sdk/amd64/2905.0.0/pkgs'
!!! HTTP Error 403: Forbidden

emerge: there are no binary packages to satisfy "sys-devel/binutils" for /build/amd64-usr/.

emerge: searching for similar names... nothing similar found.
ERROR   setup_board: script called: setup_board '--board=amd64-usr'
ERROR   setup_board: Backtrace:  (most recent call is last)
ERROR   setup_board:   file setup_board, line 330, called: die_err_trap '"${EMERGE_WRAPPER}" --usepkgonly --getbinpkg --rebuilt-binaries n ${EMERGE_TOOLCHAIN_FLAGS} "${TOOLCHAIN_PKGS[@]}"' '1'
ERROR   setup_board: 
ERROR   setup_board: Command failed:
ERROR   setup_board:   Command '"${EMERGE_WRAPPER}" --usepkgonly --getbinpkg --rebuilt-binaries n ${EMERGE_TOOLCHAIN_FLAGS} "${TOOLCHAIN_PKGS[@]}"' exited with nonzero code: 1
ERROR   setup_board:   (Note bash sometimes misreports "command not found" as exit code 1 instead of 127)

[jepio]

We are working on making this work again, but as of right now binary packages are still missing. I apologize for this.

[tbbharaj]

We are working on making this work again, but as of right now binary packages are still missing. I apologize for this.

No issues...Thanks for the update!

[SuhasreddyN]

Any update?

[jepio]

Still working on it.

[SuhasreddyN]

update?

[t-lo]

We'll provide a brief update in today's community meeting. TL;DR:

• Future releases will use https://mirror.release.flatcar-linux.net/ by default for SDK tarballs and for binary packages. This is the preferred source for building Flatcar. We're in the process of wrapping up releases to all channels - and to Mantle - which use that mirror by default; we expect to release later this week.
• For nightlies / CI builds, we introduce cached public access to the GCS bucket via https://bucket.release.flatcar-linux.net/ .

[pothos]

If you develop on the main branch in your cork SDK, run this here to get access to the nightly packages:

./set_version --binhost https://bucket.release.flatcar-linux.net/flatcar-jenkins --dev-board --board-version amd64-usr/main-nightly --dev-sdk --sdk-version sdk-main-nightly

[juliantaylor]

Is here any way to build kernel modules?
As documented here:
https://kinvolk.io/docs/flatcar-container-linux/latest/reference/developer-guides/kernel-modules/
still returns 403 for all stable versions

 !!! Error fetching binhost package info from 'https://storage.googleapis.com/flatcar-jenkins/boards/amd64-usr/2905.2.4/pkgs'
 !!! HTTP Error 403: Forbidden

[t-lo]

Because of a change in our build infrastructure the URL of our binhost has changed from

https://storage.googleapis.com/flatcar-jenkins/boards/amd64-usr/<version>/pkgs

to

https://mirror.release.flatcar-linux.net/boards/amd64-usr/<version>/pkgs

We are in the process of issuing an update to all channels to reflect the change.

[ManojMondal02062019]

We'll provide a brief update in today's community meeting. TL;DR:

• Future releases will use https://mirror.release.flatcar-linux.net/ by default for SDK tarballs and for binary packages. This is the preferred source for building Flatcar. We're in the process of wrapping up releases to all channels - and to Mantle - which use that mirror by default; we expect to release later this week.
• For nightlies / CI builds, we introduce cached public access to the GCS bucket via https://bucket.release.flatcar-linux.net/ .

What will be steps to build the Flatcar.. Meant to say that if I download the flatcar sdk from this link -> https://mirror.release.flatcar-linux.net/sdk/amd64/3005.0.0/flatcar-sdk-amd64-3005.0.0.tar.bz2 then how to proceed further with it. I extracted the file under the folder chroot and running "cork enter" - doesn't works.
@t-lo : Please suggest. Thanks

[pothos]

From yesterday on cork create will default to downloading the 3033.0.0 Alpha SDK which already has the new URLs set up for binary packages (but you need to check out the right flatcar-3033 Alpha release branches in the repos). The latest Stable, Beta, and LTS releases also have a new SDK with the right URLs (e.g., cork create --manifest-branch flatcar-stable-2905.2.6), and specifying the manifest branch will additionally check out the right release branches. Make sure to run the latest cork compiled from https://github.com/flatcar-linux/mantle

For developing on main branches after cork create, see #523 (comment)