Can't fetch private UPM Packages from github because can't add my github credentials to Unity

[Podden]

Bug description
We're using private packages from github in the Unity Package Manager. Building does not work with this because Unity asks for my github credentials. On my local Windows machine, Github Crendetial Manager pops up and I can insert them. What do I have to do in an github action to add them?

- stderr: fatal: could not read Username for 'https://github.com': terminal prompts disabled
Project has invalid dependencies:
de.lefx.stepmanager: Error when executing git command. fatal: could not read Username for 'https://github.com': terminal prompts disabled

I've tried to add my github credentials with fusion-engineering/[email protected] but does not work either.

How to reproduce

Try building a project with UPM dependency from a private github URL.

Expected behavior

Project builds :)

Additional details

I've added my script and the logfiles

build.zip
logs_51.zip

[GabLeRoux]

I'd say a workaround would be to clone these projects as part of your CI actions before building unity project.

Related question on Support community:
https://github.521000.bestmunity/t/best-way-to-clone-a-private-repo-during-script-run-of-private-github-action/16116

You'll need a personal access token generated from here. Check the Repo checkbox to access private repository. This token will grant access to what you have access to in terms of private repos.

In your action file, you can add a step like this:

- uses: actions/checkout@v1
  with:
    repository: my-private/repo-name
    token: ${{ secrets.ACCESS_TOKEN }}

Set ACCESS_TOKEN in your actions secret to the token you generated previously. ✌️

Note: I did not try this solution, but it should work unless Unity needs to do something special when adding a private UPM package.

[zZz-Theo-zZz]

Hi !

I have the exact same need as @Podden. I tried @GabLeRoux workaround by cloning dependencies directly to Unity's package cache folder but it looks like it's not working.

Dependencies referenced by the Packages/manifest.json can be written as follow :

"yourpackagename": "git+ssh://[email protected]/company/yourpackagename.git#tag-or-sha1"

This way git doesn't ask for username/password on the command line and will look for your ssh keys instead.
But it doesn't seem to work (relevant log extract below with further investigation). That being said, it could work by using FILE protocol in manifest.json (not tested yet), but it's not very flexible since you would need to update your workflow files each time you need to update your project dependencies...

Here is my log extract when trying with git+ssh in manifest.json

COMMAND LINE ARGUMENTS:
/opt/Unity/Editor/Unity
-batchmode
-logfile
/dev/stdout
-quit
-customBuildName
StandaloneWindows64
-projectPath
/github/workspace/.
-buildTarget
StandaloneWindows64
-customBuildTarget
StandaloneWindows64
-customBuildPath
/github/workspace/build/StandaloneWindows64/StandaloneWindows64.exe
-executeMethod
UnityBuilderAction.Builder.BuildProject
-buildVersion
1.1.201
-androidVersionCode
1001201
-androidKeystoreName

-androidKeystorePass

-androidKeyaliasName

-androidKeyaliasPass

-nographics
Successfully changed project path to: /github/workspace/.
/github/workspace
Using Asset Import Pipeline V2.
DisplayProgressbar: Unity Package Manager
Rebuilding Library because the asset database could not be found!
Adding genesis user token Error when executing git command:
  - stdout: 
  - stderr: No user exists for uid 1001
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Error when executing git command:
  - stdout: 
  - stderr: No user exists for uid 1001
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Googling things like 'No user exists for uid 1001' shows that it seems to be related to how docker changes user but I'm not really used to Docker and clean permission handling on linux... I tried (with no luck) adding entries in /etc/passwd as suggested here

Here is another link that seems relevant to me , but I really have no idea on how to apply that.

Here is my workflow file :

name: Build App

on:
  push: { branches: [master] }
  workflow_dispatch: { branches: [master] }

jobs:
  build:
    name: Build project
    runs-on: ubuntu-latest
    strategy:
      fail-fast: true
      matrix:
        projectPath:
          - ./
        unityVersion:
          - 2019.4.4f1
        targetPlatform:
          - StandaloneWindows64
    steps:
      # Checkout
      - name: Checkout repository
        uses: actions/checkout@v2
        with:
          lfs: false

      - name: Checkout repository
        uses: actions/checkout@v2
        with:
            token: ${{ secrets.PRIVATE_TOKEN }}
            repository: company/mypackage
            ref: 1.0.0
            path: ${{ matrix.projectPath }}/Library/PackageCache/mypackage

      # Cache
      - uses: actions/[email protected]
        with:
          path: ${{ matrix.projectPath }}/Library
          key: Library-${{ matrix.projectPath }}-${{ matrix.targetPlatform }}
          restore-keys: |
            Library-${{ matrix.projectPath }}-
            Library-

      - name: Create SSH key
        env:
          SSH_PRIVATE_KEY: ${{secrets.SSH_PRIVATE_KEY}}
          SSH_AUTH_SOCK: /tmp/ssh_agent.sock
        run: |
          sudo mkdir -p /home/github/.ssh/
          sudo ssh-keyscan github.com >> /home/github/.ssh/known_hosts
          sudo ssh-agent -a $SSH_AUTH_SOCK > /dev/null
          sudo echo $SSH_PRIVATE_KEY > /home/github/.ssh/private.key
          sudo echo "Host *" > /home/github/.ssh/config
          sudo echo "   IdentityFile /home/github/.ssh/private.key" >> /home/github/.ssh/config
          sudo chown -R runner:runner /home/github/
          sudo chmod 0600 /home/github/.ssh/private.key
        shell: bash

      # Build
      - name: Unity - Builder
        uses: webbertakken/[email protected]
        env:
          # Unity Pro Licence Activation
          # UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          # UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
          # UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
          # Unity Personal License Activation
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
          SSH_AUTH_SOCK: /tmp/ssh_agent.sock
        with:
          customParameters: -nographics
          projectPath: ${{ matrix.projectPath }}
          unityVersion: ${{ matrix.unityVersion }}
          targetPlatform: ${{ matrix.targetPlatform }}

      # Output
      - uses: actions/upload-artifact@v1
        with:
          name: Build
          path: build

[caiusno1]

@TheoRealcast maybe try again with unity-builder@main because the proper user PR was reverted because of another problem. Maybe that works ;-)

[webbertakken]

Closing this until there is more information available. Right this is probably solved in the latest version of builder.

If not, feel free to drop a comment and we'll reopen it.

[Podden]

Is there some other context I do not get?

while

      - uses: fusion-engineering/setup-git-credentials@v2'
        with:
            credentials: ${{secrets.GIT_CREDENTIALS}}
      - name: Cloning
        run: git clone https://github.com/vr-bits/UnitySensei

works flawless, this gives me the "cannot access repository" error

      - uses: fusion-engineering/setup-git-credentials@v2
        with:
            credentials: ${{secrets.GIT_CREDENTIALS}}
      - name: Build project
        uses: game-ci/[email protected]
        with:
          unityVersion: 2019.4.16f1
          targetPlatform: Android

I tried the above without the fusion-engineering/setup-git-credentials@v2, git clone gives a correct permission error so this seems to work and my credentials are correct. I'm guessing I have to inject my git credentias in the docker container in which unity runs as well? Any guesses how this can be done?

Happy new year btw :)

[webbertakken]

Currently I think the only way is to put your ssh key in ~/.ssh, which is mounted into the container.

[Podden]

Hmm, my Problem with this is on the usability side, because I have to change the manifest.json to SSH and teach all my non-dev collaborateurs how to install SSH Keys on Windows :(. I'll another way by, maybe rewriting the manifest.json in the Action.
Thanks!

[Podden]

Isn't there a point in your action to inject custom Terminal commands in the container before Unity starts?

RUN git config --global url.”https://{token}:@github.com/".insteadOf “https://github.com/"

[webbertakken]

That's a fair point and thank you for elaborating on your use case, this is very helpful.

I suppose we could add gitUser and gitPassword as parameters and pass them into the image.

I believe unity has specific ENV variables it wants to use for UPM.

Open for contributions.

[MarkyabaDev]

Does anyone work on this?

[webbertakken]

Does anyone work on this?

Feel free to take a stab at it.

[rjga94]

Currently I think the only way is to put your ssh key in ~/.ssh, which is mounted into the container.

Hi all,
@webbertakken How exactly can i do this ?

[webbertakken]

Closing this as solved by above PRs

[andreparodi-bandai]

Hi @webbertakken ,

I am in a similar situation to Podden above.

For most of my team it's easier to have https urls rather that getting ssh keys setup for everyone.

I'm interested in passing an additional 2 parameters: gitubUsername and gitubPersonalAccessToken. In addition if these variables are set I would add an insteadof command like:
RUN git config --global url.”https://{githubUsername}:{githubPersonalAccessToken}@github.com/".insteadOf “https://github.com/"

I believe this is a useful alternative to provide as it's a lot easier for users to configure git using oauth and use https. This then means you have to use Personal Tokens for CI. Also using PAT instead of GITHUB_TOKEN means you can authenticate against other repos (from I have read the scope of the GITHUB_TOKEN is repository scoped).

I am happy to have a stab at this and submit a PR if you feel there is a chance of it getting in.
Thanks
Andre

[webbertakken]

So using a token to authenticate and pull other reps sounds fine to me.

All mentioned tokens are GitHub related, private and for access, so we can simply call it githubToken.

Please note however that CI is not any user and as such shouldn't use usernames or personal access tokens in principle.

Perhaps we could add a flag named forceSshGitResolution and rewrite manifest file or within git force ssh resolution somehow, to prevent making changes on CI time.

[andreparodi-bandai]

The use case we have is as follows:
game is in private repo MYGAME
package is in private repo MYPACKAGE
both repos a private and require authentication to read.

my understanding from reading https://docs.github.com/en/actions/security-guides/automatic-token-authentication is that the githubToken provided to the context of the action is restricted to the repo in which the action is executing. Hence this token only gives me access to MYREPO. "The token's permissions are limited to the repository that contains your workflow."

Since MYPACKAGE is a different repo i'm not sure this is going to work. In te last section of the page linked above https://docs.github.com/en/actions/security-guides/automatic-token-authentication#granting-additional-permissions it suggests that if you need more permissions that are given to the githubToken you would need to create a PAT (which would need to belong to a user).

Either way, I am going to test just using githubToken and see. But my suspiscion is that if the package is in a separate private repo it won't work.

I am going to try to get it working and then see how best it can be integrated. From my current understanding there are 3 cases:

1. your packages are in the same repo as you workflow. Great! we can use the githubToken provided by github actions.
2. you want to use ssh. Great! you can sepcify the sshAgent variable and there is a solution for that (Although I got stuck on automating the known_hosts. but that's another topic)
3. you want to use https and your packages are in separate repos to your workflow. you will need to provide a PAT and username. TBC.

Andre

[webbertakken]

Since MYPACKAGE is a different repo i'm not sure this is going to work. In te last section of the page linked above https://docs.github.com/en/actions/security-guides/automatic-token-authentication#granting-additional-permissions it suggests that if you need more permissions that are given to the githubToken you would need to create a PAT (which would need to belong to a user).

I see. If that is GitHub official answer (for now) then so is ours. I stand corrected. Conceptually though not everyone has their own tokens for CI, just the person setting it up configures a token. Not sure if this is obvious for everyone or not.

As for the cases let's also consider:

4. Packages from public repositories
5. A mixture, where some might be private or even vendor owned and others public or in the same repository.

That is to say that I think we'd like to support a similar solution as the one you proposed, as long as it works for everyone or is forward compatible with all cases in some way.

Finally, let's see how much of the problem would get solved by https://github.com/game-ci/unity-builder/pull/278/files or how much of that PR can be reused.

Thank you for your extended explanation and consideration. :)

[davidmfinol]

The gitPrivateToken option has been added for access to other repos.
Between githubToken, sshAgent, and gitPrivateToken, does that cover all the cases for this issue?

[webbertakken]

Thank you @davidmfinol. Looks like it indeed.

Closing in favour of making new issues based on the latest iterations.