Passing pip arguments through on build

[oddbookworm]

What is the problem or limitation you are having?

I'm trying to build pygame-ce for android, which uses the meson-python build system. It's failing somewhere in there, but I can't figure out why. Meson-python always cleans up the temp directory it defaults to, which deletes the log it tells you to reference to see what went wrong. There's this open issue there, but it would be nice to be able to pass args directly to pip during the briefcase build stage. Specifically, the arg for this is

-Cbuild-dir=build/dir/desired

Related issue from meson-python: mesonbuild/meson-python#246

Describe the solution you'd like

I found the --config option, but it doesn't seem to like passing in something akin to the argument above. So either I'd like it to accept something like --config "-Cbuild-dir"="build/dir/desired" or some alternative way to tell briefcase to pass a specific command to pip

Describe alternatives you've considered

There's really not much I've been able to think of. Briefcase invokes pip directly internally, and this isn't currently controllable except for that argument at build time

Additional context

I suspect the issue is with dependencies not being available for android build, but without some way to get that log that gets instantly deleted, I don't know how else to confirm my suspicion.

If it's at all helpful, here's a full-verbosity log from my last build attempt

briefcase.2024_07_07-20_40_45.build.log

I'm using a custom pygame-ce build that puts android back as "supported" on the entrypoint so that it'll even attempt to build

[rmartin16]

Hello @oddbookworm,
Can you see if this workaround for passing arguments to pip works for you here? Basically, just specify the pip argument as though it was a PyPI requirement in pyproject.toml.

[oddbookworm]

That doesn't seem to have done anything

Here's what I did in pyproject.toml

requires = [
    "-Cbuild-dir=C:/Users/andre/.meson",
    "pygame-ce",
]

And here's the new log, which doesn't reflect that path at all

briefcase.2024_07_07-21_08_30.build.log

[rmartin16]

Can you try briefcase build android -vvv --update-requirements instead? That'll force running pip again during the build.

[oddbookworm]

That did result in something different, but probably not what you meant for it to do

2024-07-07T21:15:30.713-0500 [ERROR] [system.err] ERROR: Invalid requirement: 'C:\\Users\\andre\\Projects\\briefcase-android-test\\helloworld\\-Cbuild-dir=C:\\Users\\andre\\.meson' (from line 2 of requirements.txt)
2024-07-07T21:15:30.713-0500 [ERROR] [system.err] Hint: It looks like a path. File 'C:\Users\andre\Projects\briefcase-android-test\helloworld\-Cbuild-dir=C:\Users\andre\.meson' does not exist.
2024-07-07T21:15:30.752-0500 [ERROR] [system.err] Chaquopy: Exit status 1

[rmartin16]

Ok; so, for an Android Gradle build, the requirements are installed from a generated requirements.txt file. It'll just include the values in requires in pyproject.toml. So, here, this workaround depends on generating a valid requirements.txt file that contains pip arguments....off the top of my head, though, I don't remember if pip supports reading arguments from requirements.txt anymore...

[oddbookworm]

I'm not sure it's generating it correctly though... this is what build/helloworld/android/gradle/app/requirements.txt looks like

# Generated 2024-07-07 21:21:47.838778
C:\Users\andre\Projects\briefcase-android-test\helloworld\-Cbuild-dir="C:\Users\andre\.meson"
pygame-ce

[oddbookworm]

That path before the -Cbuild-dir looks sus

[oddbookworm]

I think I broke gradle lol. I tried this as the "generated" requirements.txt

# Generated 2024-07-07 21:21:47.838778
pygame-ce --install-option="-Cbuild-dir=\"C:\Users\andre\.meson\""

Now, it seems to get stuck in some kind of loop when locking and unlocking workers and the daemon registry

[rmartin16]

ahh....the format of the build-dir argument is causing it to be treated as though it's a local file path to an installable package.

This detection is based on the presence of a slash in the value for the requirement. That may make this approach untenable...

As for manually modifying the requirements.txt, that should actually work as long as you don't ever use the -r/--update-requirements flag to cause requirements.txt to be re-written. That said, the problem of formatting the content of requirements.txt still stands :)

[oddbookworm]

Lemme try something, instead of no slashes let me just use .meson as the build dir. I have no idea where it's going to try to put it though

[oddbookworm]

pyproject.toml

requires = [
    "-Cbuild-dir=\".meson\"",
    "pygame-ce",
]

hopefully the heart of the error message

2024-07-07T21:34:41.682-0500 [ERROR] [system.err] Usage: __main__.py [options]
2024-07-07T21:34:41.682-0500 [ERROR] [system.err] 
2024-07-07T21:34:41.682-0500 [ERROR] [system.err] ERROR: Invalid requirement: -Cbuild-dir=".meson"
2024-07-07T21:34:41.682-0500 [ERROR] [system.err] __main__.py: error: no such option: -C
2024-07-07T21:34:41.682-0500 [ERROR] [system.err]
2024-07-07T21:34:41.716-0500 [ERROR] [system.err] Chaquopy: Exit status 1

log
briefcase.2024_07_07-21_34_43.build.log

[oddbookworm]

Adding what lines are emitting error messages would be useful...

[rmartin16]

hmm....looks like the pip argument in requirements.txt is being read by the wrong "layer"....I'm not entirely sure which one, though.

Another option is to bypass Briefcase's abstraction of specifying requirements in pyproject.toml and just specify the requirement directly in the Gradle build configuration; there should be more options available there.

[oddbookworm]

I'm not sure if I'm doing it right, but in the provided build.gradle at build/helloworld/android/gradle/app/build.gradle, I tried two different things under the user provided content section:

chaquopy {
    defaultConfig {
        pip {
            options("-Cbuild-dir", "C:/Users/andre/.meson")
        }
    }
}

(error)

2024-07-07T22:23:09.553-0500 [ERROR] [system.err] no such option: -C
2024-07-07T22:23:09.586-0500 [ERROR] [system.err] Chaquopy: Exit status 2

and

chaquopy {
    defaultConfig {
        pip {
            options("--config-settings", "-Cbuild-dir=\"C:/Users/andre/.meson\"")
        }
    }
}

(error)

2024-07-07T22:22:15.461-0500 [ERROR] [system.err] no such option: --config-settings
2024-07-07T22:22:15.500-0500 [ERROR] [system.err] Chaquopy: Exit status 2

[rmartin16]

Chaquopy is using a vendored version of pip; it's version 20.1. Back then, looks like pip was still using --install-option.

[rmartin16]

Another option altogether here (and perhaps the more reasonable approach) is to build a wheel for pygame-ce first; then that wheel's file path can just be specified in pyproject.toml.

There are general directions at Chaquopy for building Android wheels for arbitrary packages.

[oddbookworm]

We build wheels for all sorts of systems, so maybe that would be fine. However, we build all of our wheels using CI and I don't think github CI supports android builds. IMO that's not a very sustainable approach for us to take because we'd need to host the build somewhere and ensure that someone builds it regularly by hand. I am going to tag in @ankith26 though, who is our "build system and CI guru"

[oddbookworm]

Chaquopy is using a vendored version of pip; it's version 20.1. Back then, looks like pip was still using --install-option.

Using this

chaquopy {
    defaultConfig {
        pip {
            options("--install-option", "-Cbuild-dir=C:/Users/andre/.meson")
        }
    }
}

Gave me the same "gradle infinite loop" as when I tried that same thing before here

[ankith26]

However, we build all of our wheels using CI and I don't think github CI supports android builds.

If we figure out the process, setting up CI to repeat the process shouldn't be hard. As for passing arguments to meson, you can always edit pyproject.tomls [tool.meson-python.args] section

[oddbookworm]

If we figure out the process, setting up CI to repeat the process shouldn't be hard. As for passing arguments to meson, you can always edit pyproject.tomls [tool.meson-python.args] section

This caused the same infinite looping problem, but I thought to try to use pygame-ce 2.4.1 instead of latest. 2.4.1 was before we migrated to meson and uses the old setup.py build process. Turns out, we're not getting a useful compiler from chaquopy

[oddbookworm]

2024-07-08T22:09:58.293-0500 [QUIET] [system.out]   Building wheel for pygame-ce (setup.py): finished with status 'error'
2024-07-08T22:09:58.293-0500 [ERROR] [system.err]   ERROR: Command errored out with exit status 1:
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]    command: 'C:\Users\andre\Projects\briefcase-android-test\helloworld\build\helloworld\android\gradle\app\build\python\env\debug\Scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\andre\\AppData\\Local\\Temp\\pip-install-rk4_7lsr\\pygame-ce\\setup.py'"'"'; __file__='"'"'C:\\Users\\andre\\AppData\\Local\\Temp\\pip-install-rk4_7lsr\\pygame-ce\\setup.py'"'"';import chaquopy_monkey; chaquopy_monkey.disable_native();f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\andre\AppData\Local\Temp\pip-wheel-s3wjdn8b'
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]        cwd: C:\Users\andre\AppData\Local\Temp\pip-install-rk4_7lsr\pygame-ce\
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]   Complete output (9 lines):
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]   ---
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]   For help with compilation see:
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]       https://github.com/pygame-community/pygame-ce/wiki/Compiling-on-Windows
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]   To contribute to pygame-ce development see:
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]       https://github.com/pygame-community/pygame-ce
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]   ---
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]   error: CCompiler.compile: Chaquopy cannot compile native code
2024-07-08T22:09:58.294-0500 [ERROR] [system.err]   ----------------------------------------
2024-07-08T22:09:58.294-0500 [ERROR] [system.err] ERROR: Failed to install pygame-ce<2.5 from https://files.pythonhosted.org/packages/dc/b1/64fffc2c8664497ae82b2afb4f5efe0130d38b39f2d25af0288c4261df3e/pygame-ce-2.4.1.tar.gz#sha256=70a84aa1417c633a0fd6754ffa5dc92ee1b9aeb70baaa52c8c8c94a7c6db9cf0 (from -r requirements.txt (line 2)).
2024-07-08T22:09:58.295-0500 [ERROR] [system.err] For assistance, please raise an issue at https://github.com/chaquo/chaquopy/issues.
2024-07-08T22:09:58.460-0500 [ERROR] [system.err] Chaquopy: Exit status 1
2024-07-08T22:09:58.483-0500 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: FAILED
2024-07-08T22:09:58.483-0500 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Process 'command 'C:\Users\andre\Projects\briefcase-android-test\helloworld\build\helloworld\android\gradle\app\build\python\env\debug\Scripts\python.exe'' finished with exit value 1 (state: FAILED)

[oddbookworm]

I guess we are just going to have to build our own wheels
I'll try to look into that over the next few days

[mhsmith]

IMO that's not a very sustainable approach for us to take because we'd need to host the build somewhere

We already have a repository for Android wheels, and Briefcase uses it automatically when building Android apps. If you follow these instructions to create a recipe, and make a PR in the Chaquopy repository, we'll happily host the wheels for you.

[oddbookworm]

(Moved to chaquo/chaquopy#1278)

[mhsmith]

Let's keep this page focused on the pip arguments issue, and discuss pygame-ce specifically in chaquo/chaquopy#1278.