Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: upgrade underlying debian version #11472

Merged
merged 1 commit into from
Jun 1, 2022

Conversation

miketheman
Copy link
Member

Debian stable has moved from buster (10) to bullseye (11).

Swapping out the underlying Debian version provides us with acces to
newer package dependencies, specifically: gcc 8.3 => 10.2

Refs: https://www.debian.org/releases/bullseye/
Refs: https://www.debian.org/releases/bullseye/amd64/release-notes/ch-whats-new.en.html#newdistro

Signed-off-by: Mike Fiedler [email protected]

Debian `stable` has moved from buster (10) to bullseye (11).

Swapping out the underlying Debian version provides us with acces to
newer package dependencies, specifically: gcc 8.3 => 10.2

Refs: https://www.debian.org/releases/bullseye/
Refs: https://www.debian.org/releases/bullseye/amd64/release-notes/ch-whats-new.en.html#newdistro

Signed-off-by: Mike Fiedler <[email protected]>
Copy link
Member

@alex alex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Will wait for another warehouse dev who is actually involved in ops to merge though.

@miketheman
Copy link
Member Author

Thanks @alex ! That should probably be @ewdurbin or @di I guess, unless there's others that hold that honor?

Copy link
Member

@di di left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will let @ewdurbin merge this.

@ewdurbin ewdurbin merged commit 8406057 into pypi:main Jun 1, 2022
@miketheman miketheman deleted the miketheman/upgrade-debian branch June 1, 2022 17:17
@ewdurbin
Copy link
Member

ewdurbin commented Jun 1, 2022

Interestingly this fails to build the image in cabotage (running on linux) but is fine locally for me (MacOS arm64).

See imagemin/imagemin-gifsicle#37 (comment)

Output from cabotage build:

Step 1/32 : FROM node:14.19.1-bullseye as static
14.19.1-bullseye: Pulling from library/node
67e8aa6c8bbc: Pulling fs layer
627e6c1e1055: Pulling fs layer
0670968926f6: Pulling fs layer
5a8b0e20be4b: Pulling fs layer
b0b10a3a2784: Pulling fs layer
fb13efbeef09: Pulling fs layer
85b2df93731e: Pulling fs layer
a08f2c0b344b: Pulling fs layer
8b33cce68368: Pulling fs layer
5a8b0e20be4b: Download complete
8b33cce68368: Waiting
fb13efbeef09: Download complete
627e6c1e1055: Download complete
0670968926f6: Download complete
a08f2c0b344b: Download complete
85b2df93731e: Verifying Checksum
85b2df93731e: Download complete
8b33cce68368: Verifying Checksum
8b33cce68368: Download complete
b0b10a3a2784: Verifying Checksum
b0b10a3a2784: Download complete
67e8aa6c8bbc: Pull complete
627e6c1e1055: Pull complete
0670968926f6: Pull complete
5a8b0e20be4b: Pull complete
b0b10a3a2784: Pull complete
fb13efbeef09: Pull complete
85b2df93731e: Pull complete
a08f2c0b344b: Pull complete
8b33cce68368: Pull complete
Digest: sha256:194587b204eb3f8576f0a40a8cd80a194c4d44c965b0c1bd7dfc3279fb7c2f06
Status: Downloaded newer image for node:14.19.1-bullseye
 ---> 63fe48818635
Step 2/32 : WORKDIR /opt/warehouse/src/
Removing intermediate container d0106c12f493
 ---> ecd2117c5a99
Step 3/32 : RUN set -x     && apt-get update     && apt-get install --no-install-recommends -y         libjpeg-dev nasm
 ---> Running in 344e437062d2
�[91m+ apt-get update
�[0mGet:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB]
Get:4 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [153 kB]
Get:5 http://deb.debian.org/debian bullseye/main amd64 Packages [8182 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2592 B]
Fetched 8537 kB in 2s (4878 kB/s)
Reading package lists...
�[91m+ apt-get install --no-install-recommends -y libjpeg-dev nasm
�[0mReading package lists...
Building dependency tree...
Reading state information...
libjpeg-dev is already the newest version (1:2.0.6-4).
The following NEW packages will be installed:
  nasm
0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
Need to get 421 kB of archives.
After this operation, 3378 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main amd64 nasm amd64 2.15.05-1 [421 kB]
�[91mdebconf: delaying package configuration, since apt-utils is not installed
�[0mFetched 421 kB in 0s (4111 kB/s)
Selecting previously unselected package nasm.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 22741 files and directories currently installed.)
Preparing to unpack .../nasm_2.15.05-1_amd64.deb ...
Unpacking nasm (2.15.05-1) ...
Setting up nasm (2.15.05-1) ...
Removing intermediate container 344e437062d2
 ---> 1b3a4e6d546a
Step 4/32 : COPY package.json package-lock.json .babelrc /opt/warehouse/src/
 ---> 6bdb97601a7f
Step 5/32 : RUN set -x     && npm install -g gulp-cli     && export CFLAGS="-DPNG_ARM_NEON_OPT=0" && npm ci
 ---> Running in 3bac1eae0625
�[91m+ npm install -g gulp-cli
�[0m�[91mnpm WARN deprecated [email protected]: See https://github.com/lydell/source-map-resolve#deprecated
�[0m�[91mnpm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
�[0m�[91mnpm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
�[0m�[91mnpm WARN deprecated�[0m�[91m [email protected]: See https://github.com/lydell/source-map-url#deprecated
�[0m/usr/local/bin/gulp -> /usr/local/lib/node_modules/gulp-cli/bin/gulp.js

> [email protected] postinstall /usr/local/lib/node_modules/gulp-cli/node_modules/es5-ext
>  node -e "try{require('./_postinstall')}catch(e){}" || exit 0

+ [email protected]
added 254 packages from 165 contributors in 15.776s
�[91m+ export CFLAGS=-DPNG_ARM_NEON_OPT=0
+ npm ci
�[0m
> [email protected] postinstall /opt/warehouse/src/node_modules/zopflipng-bin
> node lib/install.js

�[91m  ✔ zopflipng pre-build test passed successfully
�[0m
> [email protected] install /opt/warehouse/src/node_modules/fsevents
> node install


> [email protected] postinstall /opt/warehouse/src/node_modules/pngquant-bin
> node lib/install.js

�[91m  ✔ pngquant pre-build test passed successfully
�[0m
> [email protected] postinstall /opt/warehouse/src/node_modules/optipng-bin
> node lib/install.js

�[91m  ✔ optipng pre-build test passed successfully
�[0m
> [email protected] postinstall /opt/warehouse/src/node_modules/mozjpeg
> node lib/install.js

�[91m  ✔ mozjpeg pre-build test passed successfully
�[0m
> [email protected] postinstall /opt/warehouse/src/node_modules/jpeg-recompress-bin
> node lib/install.js

�[91m  ✔ jpeg-recompress pre-build test passed successfully
�[0m
> [email protected] postinstall /opt/warehouse/src/node_modules/guetzli
> node lib/install.js

�[91m  ⚠ Command failed: /opt/warehouse/src/node_modules/guetzli/vendor/guetzli --verbose
Guetzli JPEG compressor. Usage: 
guetzli [flags] input_filename output_filename

Flags:
  --verbose    - Print a verbose trace of all attempts to standard output.
  --quality Q  - Visual quality to aim for, expressed as a JPEG quality value.
  --memlimit M - Memory limit in MB. Guetzli will fail if unable to stay under
                 the limit. Default is 6000 MB
  --nomemlimit - Do not limit memory usage.


  ⚠ guetzli pre-build test failed
�[0m�[91m  ℹ compiling from source
�[0m�[91m  ✔ guetzli built successfully
�[0m
> [email protected] postinstall /opt/warehouse/src/node_modules/gifsicle
> node lib/install.js

�[91m  ⚠ Response code 404 (Not Found)
  ⚠ gifsicle pre-build test failed
�[0m�[91m  ℹ compiling from source
�[0m�[91m  ✖ Error: Command failed: /bin/sh -c ./configure --disable-gifview --disable-gifdiff --prefix="/opt/warehouse/src/node_modules/gifsicle/vendor" --bindir="/opt/warehouse/src/node_modules/gifsicle/vendor"
ls: cannot access '.': Operation not permitted
configure: error: working directory cannot be determined


    at /opt/warehouse/src/node_modules/execa/index.js:231:11
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Promise.all (index 0)
�[0m�[91mnpm ERR! code ELIFECYCLE
�[0m�[91mnpm ERR! errno 1
�[0m�[91mnpm ERR! [email protected] postinstall: `node lib/install.js`
npm ERR! Exit status 1
�[0m�[91mnpm ERR!�[0m�[91m 
npm ERR!�[0m�[91m Failed at the [email protected] postinstall script.
npm �[0m�[91mERR!�[0m�[91m This is probably not a problem with npm. There is likely additional logging output above.
�[0m�[91m
npm �[0m�[91mERR! A complete log of this run can be found in:
npm �[0m�[91mERR!     /root/.npm/_logs/2022-06-01T15_41_33_866Z-debug.log
�[0mRemoving intermediate container 3bac1eae0625

ewdurbin added a commit that referenced this pull request Jun 1, 2022
@miketheman
Copy link
Member Author

Thanks for the fix - I guess this points out a weakness in the current CI steps, as they don't do the same thing as cabotage did, which is build the Docker image. Should the CI steps do that as well?

@ewdurbin
Copy link
Member

ewdurbin commented Jun 1, 2022

Likely yes. When I can figure out what the difference is between the docker builds that causes this. It doesn't do much good to just run docker build . it seems, since I did that, it succeeded, and it obscured an issue.

ewdurbin added a commit that referenced this pull request Jun 1, 2022
* Revert "Fix the production container build for new bullseye base images (#11492)"

This reverts commit 1671a69.

* Revert "chore: upgrade underlying debian version (#11472)"

This reverts commit 8406057.
@miketheman miketheman restored the miketheman/upgrade-debian branch June 2, 2022 23:05
@miketheman
Copy link
Member Author

@ewdurbin I was trying this again, and can't seem to repro on a linux system, but I was running make build. Where might I find what cabotage runs, or its execution environment to help make sense of this issue?

domdfcoding pushed a commit to domdfcoding/warehouse that referenced this pull request Jun 7, 2022
Debian `stable` has moved from buster (10) to bullseye (11).

Swapping out the underlying Debian version provides us with acces to
newer package dependencies, specifically: gcc 8.3 => 10.2

Refs: https://www.debian.org/releases/bullseye/
Refs: https://www.debian.org/releases/bullseye/amd64/release-notes/ch-whats-new.en.html#newdistro

Signed-off-by: Mike Fiedler <[email protected]>
domdfcoding pushed a commit to domdfcoding/warehouse that referenced this pull request Jun 7, 2022
* Revert "Fix the production container build for new bullseye base images (pypi#11492)"

This reverts commit 1671a69.

* Revert "chore: upgrade underlying debian version (pypi#11472)"

This reverts commit 8406057.
@miketheman miketheman deleted the miketheman/upgrade-debian branch June 17, 2022 15:45
@miketheman
Copy link
Member Author

Circling back to this, I still haven't been able to reproduce the error observed.

I'm inclined to try two actions here:

  1. Update the underlying pinned gifsicle version from 5.0.0 to 5.3.0. Still compatible with gulp-image since that version don't seem to need compilation on buster or bullseye, sidestepping the issue
  2. Update only the static image to bullseye isolate variables, since that's the one showing an error

I'll open up a new PR and reference this one.

miketheman added a commit to miketheman/warehouse that referenced this pull request Sep 24, 2022
As previously attempted in pypi#11472 and reverted in pypi#11493

After isolating the condition that was observed and removing the calls
to compile `gifsicle` in pypi#12232, upgrade the `static` containers only
with this changest.

Signed-off-by: Mike Fiedler <[email protected]>
ewdurbin pushed a commit that referenced this pull request Sep 26, 2022
As previously attempted in #11472 and reverted in #11493

After isolating the condition that was observed and removing the calls
to compile `gifsicle` in #12232, upgrade the `static` containers only
with this changest.

Signed-off-by: Mike Fiedler <[email protected]>

Signed-off-by: Mike Fiedler <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants