Unable to configure scancode on a macOS jenkins server

[JoeSzymanski]

Description

When attempting to use scancode on a macOS jenkins server, the configure step fails because of errors attempting to install the scancode wheel.

WARNING: Requirement 'scancode_toolkit*.whl' looks like a filename, but the file does not exist ERROR: scancode_toolkit*.whl is not a valid wheel filename.

From what I can tell, the glob is not being exanded properly. When testing locally, or when testing via SSH into the same server, the command works as expected.

We have tried using set +o noglob to force the glob to expand, but that doesn't appear to work because of the subshells generated by the configure script. We have also tried forcing the script to run in a new bash shell via /bin/bash -xe ./configure and arch -x86_64 /bin/bash -xe ./configure, but all options fail with the same error.

How To Reproduce

In a jenkins script, run scancode commands to trigger the configure script.

System configuration

Running macOS Monterey (12.6) via a jenkins server. This happens on both Intel and M1 based servers.

[pombredanne]

@JoeSzymanski Thanks for the report... can you provide a few more details on how you install scancode?
Which version? Fetched from where? How do you install it? And with which Python version?

[JoeSzymanski]

We are downloading the packaged macOS version of v31.1.1 (https://github.com/nexB/scancode-toolkit/releases/download/v31.1.1/scancode-toolkit-v31.1.1_py3.8-macos.tar.gz) and unzipping that into a standard directory.

We have the system version of Python installed, though to my knowledge, the packages cancode releases include an internal version of Python so that there aren't issues with the system versions.

Once we have the packages xcancode unzipped, we just call either the scancode command or directly call the configure command. In both cases, we hit the same error where it appears the glob is not expanded.

[pombredanne]

We run extra release smoke tests on macOS 12 here https://github.com/nexB/scancode-toolkit/actions/runs/2978733053/jobs/4773993057 for instance and this runs this plain configure .... https://github.com/nexB/scancode-toolkit/blob/develop/etc/release/scancode_release_tests.py

[pombredanne]

@JoeSzymanski re: #3108 (comment)

Thanks for the details! That's weird. It this something that happens only in the Jenkins server? Would you have by chance another mac to try this not under Jenkins?

[pombredanne]

We have the system version of Python installed, though to my knowledge, the packages cancode releases include an internal version of Python so that there aren't issues with the system versions.

we are isolated from the python libraries, but we use the system python for now, not an internal version (yet that's in the plan for the future)

[JoeSzymanski]

I've tried running the command on my own Mac, and on the same Mac instance used by the jenkins server (via SSH and a terminal instance over Remote Desktop). In all other instances, the commands work correctly. It's only when running the command inside a jenkins job that the commands fail.

That's partly why we tried forcing the command to be run inside a bash shell instance (with /bin/bash ./configure), as we weren't sure if the shell used by jenkins was causing problems, but that didn't appear to help at all.

[pombredanne]

@JoeSzymanski OK, what would be the simplest way to try to reproduce this? knowing that I could run something on a mac from Azure or GitHub actions? (Or I can may be fine someones running a mac, I run on Linux otherwise)

[JoeSzymanski]

Give me some time to think that over. We still aren't sure of the exact cause, and I'm not sure how easily you could find or setup a jenkins server (especially since I'm not sure if this is a Mac specific issue or not).

I'm going to see if a coworker can help test on a non-mac jenkins instance and I will update here once we get time to figure that out.

Barring that, I might see if I can figure out how to confirm the full environment, including all environment variables, the shell in use, and the shell settings.

[JoeSzymanski]

I've been doing more digging today, and I can't find any specific reason for the failures when running within jenkins.

I was able to dump as much information as I could find around the environment: shell type, env vars, bash options. When comparing between the jenkins job instance and the ssh instance, the only difference I was able to see that I thought would impact this issue was that the jenkins jobs was not running a login shell. However, when I forced the script to run as a login shell (or forced the command to run inside a separate bash instance running a login shell), it still failed with the same error.

If I run the command ls *.whl from the scancode folder before running the configure command, the glob is expanded as expected.

Separately, I realized that I can't easily test this on non-mac devices, because I'm using the macOS version of the tool. I'm not sure if testing with other platforms would work because of the change in platform, or because of the change in the package version.

I'm running out of further ideas on what else to troubleshoot here to try to track down the issue.

[pombredanne]

re: macOS is a rather peculiar beast and has specific failure modes ;) AFAIK, Jenkits being mostly Java may add another layer of issues ... I managed to borrow a macBook M1 and another one using Intel for a couple days. Both run macOS 12.x.... if you can provide reproducible instructions to install and setup something that fails consistently that would be great

I also found this: https://support.prodi.gy/t/prodigy-whl-is-not-a-valid-wheel-filename/4278 ... the issue is clearly the lack of shell expansion IMHO... and there could be a difference on how this is handled in an interactive login vs. a non interactive one possibly compounded by zsh vs. bash behaviours.

Now we could also find a better way than to use shell expansion when doing this installation which would bypass the root of the issue entirely!
This line https://github.com/nexB/scancode-toolkit/blob/6007301cca4eff424202abf197467ee2f004e139/configure#L156

... is just trying to get the name of the one wheel file named this way in the application archive.

This was designed to be working without changing the script when the version is bumped up.
A possible better approach could be to just use the real wheel filename built during the release script https://github.com/nexB/scancode-toolkit/blob/6007301cca4eff424202abf197467ee2f004e139/.github/workflows/scancode-release.yml#L138 that calls this etc/release/scancode-create-release-app-macos.sh

https://github.com/nexB/scancode-toolkit/blob/6007301cca4eff424202abf197467ee2f004e139/etc/release/scancode-create-release-app-macos.sh could very much mutate/update the configure script with the actual built wheel filename either:

• create a symlink from scancode_toolkit.whl to the actual wheel
• hardcode the name of the released wheel in the configure script at https://github.com/nexB/scancode-toolkit/blob/6007301cca4eff424202abf197467ee2f004e139/configure#L156 with some placeholder and a sed call or similar (or a Python script)

[JoeSzymanski]

I will spend some time next week to see if I can get a stand-alone jenkins instance running locally on my Mac laptop and if I can reproduce the issue. That should help testing here and make it clear whether it’s a wider issue or not.

[pombredanne]

@JoeSzymanski Thanks! mucho appreciated.

[JoeSzymanski]

I was able to get this running locally on a jenkins instance and was able to replicate, so this should allow testing of what's going on more widely. Here are replication steps for everything:

• Following instructions from https://phoenixnap.com/kb/install-jenkins-on-mac
• Install Homebrew
• Install Java
  • brew install openjdk@11
  • sudo ln -sfn /opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-11.jdk
  • echo 'export PATH="/opt/homebrew/opt/openjdk@11/bin:$PATH"' >> ~/.zshrc
• Install Jenkins
  • brew install jenkins-lts
  • brew services start jenkins-lts
• Configure Jenkins
  • Navigate to http://localhost:8080/
  • Get Password from ~/.jenkins/secrets/initialAdminPassword and enter it to unlock Jenkins
  • Install suggested plugins
  • Create new user and leave default URL
• Download Scancode v31.1.1 for macOS into a known location
• Create a Test Job
  • Click “Create a job”
  • Enter a name like “test-scancode-configure” and select Freestyle Project then click to continue
  • Click “Add build step” and choose “Execute Shell”
  • In the script, run the command “/path/to/scancode/configure” and click save
  • Click “Build Now” on the left side menu
  • Click on the scheduled job on the left side, then click Console Output on the left side menu of the next page
  • Note the job fails with the glob error
• To run the job again, delete the folder “/path/to/scancode/venv”

[JoeSzymanski]

@pombredanne Any updates here? We've been blocked on usage of this tool because of this bug.

[JoeSzymanski]

Just as a confirmation, the issue still exists in the recently release v31.2.1

[bwjohnson-ss]

I just ran into this same issue with v31.2.6

[pombredanne]

@bwjohnson-ss thank you for the report! ... could you check the https://github.com/nexB/scancode-toolkit/releases/tag/v32.0.0rc4 ?

[enricozb]

Here's a dockerfile that reproduces the error:

FROM python:alpine3.18

RUN apk update && apk add --no-cache bash curl unzip

WORKDIR /dependencies

ARG SCANCODE_VERSION=32.0.0rc4

RUN curl -L \
    https://github.com/nexB/scancode-toolkit/releases/download/v${SCANCODE_VERSION}/scancode-toolkit-v${SCANCODE_VERSION}_py3.8-linux.tar.gz \
    --output scancode.tar.gz

RUN tar -xzf scancode.tar.gz
RUN echo "$(which python)" > scancode-toolkit-v${SCANCODE_VERSION}/PYTHON_EXECUTABLE
RUN scancode-toolkit-v${SCANCODE_VERSION}/configure

I've tried with a few different scancode versions, and I keep getting the error:

Step 8/8 : RUN scancode-toolkit-v${SCANCODE_VERSION}/configure
 ---> Running in baec92a45639
created virtual environment CPython3.11.3.final.0-64 in 248ms
  creator CPython3Posix(dest=/dependencies/scancode-toolkit-v32.0.0rc4/venv, clear=False, no_vcs_ignore=True, global=False)
  seeder FromAppData(download=False, pip=embed, setuptools=embed, wheel=embed, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
WARNING: Requirement 'scancode_toolkit*.whl' looks like a filename, but the file does not exist
ERROR: scancode_toolkit*.whl is not a valid wheel filename.
The command '/bin/sh -c scancode-toolkit-v${SCANCODE_VERSION}/configure' returned a non-zero code: 1
* Configuring ScanCode for first use...
ERROR: scancode_toolkit*.whl is not a valid wheel filename.

[pombredanne]

@enricozb thanks for the report, but yours is likely a different issue. I am not sure what is the real issue but we do not support Alpine Linux as base images. It has to be a Debian/Ubuntu or Fedora/RHEL base and only on X86/64 for now, otherwise you need to be ready to recompile all dependencies that contains native binaries compiled from C/C++ code.

[enricozb]

Good to know that Alpine isn't supported. I was actually able to get past this error by CD'ing into the configure directory before running it. The glob pattern scancode_toolkit*.whl won't expand if there aren't any matching files. That might be the issue here.

[pombredanne]

@enricozb Good that you found the way! Do you mind to share this updated Dockerfile?

And re:

Good to know that Alpine isn't supported.

The pre-built wheels and binaries we use and provide on PyPI are all built using glibc... so this negates a bit any benefits from Alpine, short of rebuilding everything first.

[enricozb]

FROM python:3.8-buster

RUN apt-get -y update && apt-get -y upgrade
RUN apt-get -y install curl unzip

# ---------- dependencies ----------
WORKDIR /dependencies

# scancode: https://github.com/nexB/scancode-toolkit
ARG SCANCODE_VERSION=31.2.6

RUN apt-get -y install python3-dev libbz2-1.0 xz-utils zlib1g libxml2-dev libxslt1-dev libpopt0

RUN curl -L \
    https://github.com/nexB/scancode-toolkit/releases/download/v${SCANCODE_VERSION}/scancode-toolkit-v${SCANCODE_VERSION}_py3.8-linux.tar.gz \
    --output scancode.tar.gz

RUN tar -xzf scancode.tar.gz
RUN echo "$(which python)" > scancode-toolkit-v${SCANCODE_VERSION}/PYTHON_EXECUTABLE
RUN cd scancode-toolkit-v${SCANCODE_VERSION}/ && ./configure

[pombredanne]

@enricozb and this is working, correct?

[enricozb]

Yep