Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update publish-pypi.md with consistent usage for "test PyPI". #224

Merged
merged 4 commits into from
Apr 15, 2024
Merged

Update publish-pypi.md with consistent usage for "test PyPI". #224

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a99d231
Update publish-pypi.md
ptressel Apr 8, 2024
fe7a0e6
Standardize on TestPyPI to refer to the test PyPI repository.
ptressel Apr 10, 2024
223b992
Fix changes requested in review.
ptressel Apr 10, 2024
2e7e3da
Remove completed todo tasks.
ptressel Apr 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 54 additions & 72 deletions tutorials/publish-pypi.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,7 @@
# Publish your Python package to PyPI

:::{todo}
- emphasize that we recommended the trusted publisher GitHub action for most maintainers
- Make sure they add /dist to their .gitignore file. We have not discussed GitHub workflows anywhere yet. Where does that fit?
- https://hatch.pypa.io/latest/intro/#existing-project <- hatch will migrate from setup.py for you - if we go with hatch then we may want to add this to the installable code lesson
:::


```bash
pipx install hatch
installed package hatch 1.9.1, installed using Python 3.12.1
These apps are now globally available
- hatch
done! ✨ 🌟 ✨
```

:::

Expand All @@ -28,48 +16,46 @@ In the previous Python packaging lessons, you've learned:
In this lesson you will learn how to:

- Build your package's source (sdist) and wheel distributions
- Setup an account on test PyPI (the process is similar for the real PyPI)
- Publish your package to test PyPI
- Setup an account on TestPyPI (the process is similar for PyPI)
- Publish your package to TestPyPI and PyPI

You will do all of your development work in this lesson using [Hatch](https://hatch.pypa.io/latest/).

Once your package is on PyPI you can publish it to conda-forge (which is a channel on conda)
using [Grayskull](https://conda.github.io/grayskull/).

You will learn how to publish to conda-forge in a future lesson.

You will learn how to publish to conda-forge in the [next lesson](publish-conda-forge).

:::
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the above docs section being added to the TODO? This will prevent the content from rendering. Might be a mistake.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look at the current rendered page. The code block about hatch is spurious -- it does not belong in the visible text there. It just appears out of nowhere, prior to the start of the meaningful text in the page. Rather, it is part of the "todo" section, and is referred to by the "todo."

Also, if you look at the current raw text, you will see that the triple-colon markup did not match opening and closing triple-colon marks. There was one extra, in the middle of the "todo" section, and one missing, where the end of the first "admonition" should have been (going by how that is structured in other pages, where the leading "workflow" image is not inside the "admonition").

I am guessing that @ucodery is looking at the "Preview", and seeing the visible triple-colon marks...? Look at other unmodified pages using "Preview", and they will be visible there as well.

This brings up a more general issue. When we work on pages on GitHub, and view the "Preview", that apparently does not run whatever markdown extensions are in use. In particular, the rendering on GitHub Preview does not know about the triple-colon markup, so just treats it as plain text.

So...maybe we should be looking into a way to do the full rendering as part of testing. That might mean doing things the Old Way, with folks doing command-line git on their local machines, and doing the rendering there, so as to see what the finished product will look like. That is significantly higher friction, but if it resolves questions like this, maybe it's the right thing to do.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note the "todo" may be obsolete -- some of those may be "done". ;-). The relevant line in the "todo" is:

"if we go with hatch then we may want to add this to the installable code lesson"

followed by the hatch code block to add to the "installable code lesson". That's already in, right at the top of the "Install hatch" section in the hatch page, so it looks like at least part of that "todo" is "done".

Aaaand, this brings up another issue: Should responses to commands be shown? And if so, how should they be formatted? The "pipx install hatch" command in this page includes the response, where the one over in the hatch page does not. Which is less confusing? Is it useful for folks to know what a correct response looks like, as opposed to an error message? There are other examples of commands, both with and without visible responses, and the formatting of the response is not consistent. Some have "#" before the response, others don't.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are right, I wasn't looking at the html rendered docs. I got confused matching up ::: myself, but this renders great, thanks for cleaning this up. However, I don't think we need to keep CLI examples around in TODOs. The install steps as well as the 3rd bullet can be dropped entirely from the TODO as they are already addressed elsewhere.

As for github's less-than rendering, I'm not sure there is anything we can do. this guide is using MyST which is essentially a mark up transpiler. It reads fully valid markdown files, which are tagged as such in the file name, so github is doing the best it can.

As for command line output - this is a very tricky question. I can see now that we do it all sorts of ways. I'm not aware of any documentation out there that gets it "right" all of the time. Partly because ```bash might be capturing a pure bash script, or it might be copy from the terminal, which is mixing stdin, stdout, stderr. I sometimes have the same issue with Python, where I want to show the repr output as in the interactive shell. Personally I try to mark the command section of interactive code examples with >>> in Python and $ in Bash, but even still I am inconsistent, even with my use of comments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was meaning to inquire... There are a lot of hatch commands here, that maybe could go in the hatch tutorial, if it would be useful to have that be a somewhat standalone guide to hatch, rather than setup / installation of hatch for later use in other tutorials. Even if these do go in the hatch tutorial, it could still be useful to repeat them here, with the specific arguments that apply to publishing to TestPyPI. I gather that the separate hatch tutorial was more recent than the use of hatch in the other tutorials.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, 1st and 3rd todos are removed. A note about converting setup.py is already in the Hatch tutorial.

About command line stuff... I was hesitating to bring this up, but... There are a fair number of folks using Windows. The commands may look a bit different if executed in cmd...but not always. A fair number of tools that come in from the *x world will use forward slashes in paths, or not understand a drive letter, or require their own special syntax for that. Others will take back slashes, but need them quoted. The best ones know what OS they're running on, and do "when in Rome...".

Windows users could install Git Bash. Or Windows Subsystem for Linux (WSL; which allows having an entire Linux distro under Windows). Or the classic Cygwin. Or adapt the commands for running directly on Windows. The easiest option is likely Git Bash. It's not entirely friction-free, however, as Git Bash's executable path may differ from the Windows path.

I've mostly ended up modifying commands so they work on cmd, except when the documentation strongly encourages use of Git Bash. Or when there's a canned development environment that can be run on a virtual machine. That's what we used to do for an open-source emergency management tool I used to work on, when we participated in hackathons. It could take the entire day to set up our environment, so instead, we prepared pre-configured virtual machine images.

The following has a good list of the options, one could just drop in a reference to this at the outset, wherever it says "we don't support Windows". :D :D :D

https://itsfoss.com/run-linux-commands-in-windows/

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising this @ptressel. We do think about OS inter-operability deeply sometimes. Unfortunately we don't pay attention to it nearly often enough, and probably even less do we test what we recommend on Windows.

#106 (comment)
#106 (comment)
#120 (comment)

This is a really big topic thought, so I have opened a new issue #240 so we don't derail your contribution.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi all 👋 it doesn't surprise me that there are some todo's left (likely by me) in the text. And also because i don't have windows, i don't catch windows-specific issues BUT we can add tabs for commands that might look different in a windows shell if needed. i think @ucodery opening #240 is perfect. my hope is that over time as more people test these out we can refine and make sure more use cases are covered.


:::{figure-md} build-workflow-tutorial
<img src="../images/tutorials/publish-package-pypi-conda.png" alt="Graphic showing the high level packaging workflow. On the left you see a graphic with code, metadata and tests in it. Those items all go into your package. An arrow to the right takes you to a build distribution files box. Another arrow to the right takes you to a publish to PyPI box which has an arrow containing sdist and wheel that notes those files go to PyPI for hosting. From PyPI is an arrow containing sdist since you can then connect to conda-forge for an automated build that sends distributions from PyPI to conda-forge." width="700px">

You need to build your Python package in order to publish it to PyPI (or Conda). The build process organizes your code and metadata into a distribution format that can be uploaded to PyPI and subsequently downloaded and installed by users.
:::

## Test PyPI vs PyPI
## TestPyPI vs PyPI

There are two "warehouses" that you can use to publish
There are two repositories associated with PyPI to which you can upload
your Python package.

1. **[Test PyPI](https://test.pypi.org):** Test PyPI is a version of the PyPI repository that can be used for testing. This is a great place to practice and learn how to publish a package without taking up space on the real PyPI servers.
2. **[Real PyPI](https://pypi.org):** This is the PyPI "warehouse" where you can officially publish your Python package. IMPORTANT: only publish your package to PyPI when you are ready for it to be used by others and/or confident that it will become a package that you maintain. PyPI is not a place to practice learning how to publish a Python package.
1. **[TestPyPI](https://test.pypi.org):** TestPyPI is a package repository provided by PyPI that you can use for testing that your package can be uploaded, downloaded, and installed correctly. This is a great place to practice and learn how to publish a package without exposing your incomplete package on the real PyPI service.
2. **[PyPI](https://pypi.org):** This is the live, production PyPI repository where you can officially publish your Python package, and from which users will get your package. IMPORTANT: Only publish your package to PyPI when you are ready for it to be used by others and/or confident that it will become a package that you will maintain. PyPI is not a place to practice learning how to publish a Python package.

The steps for publishing on test PyPI vs. real PyPI are the same with the
exception of a different url. Thus, in this lesson you will use test PyPI
to practice and learn.
The steps for publishing on TestPyPI vs. PyPI are similar with the
exception of a different url. We will point out where they differ.

## 4 Steps for publishing a Python package on PyPI
## 4 Steps for publishing a Python package on TestPyPI (or PyPI)

In this lesson you will learn how to publish your package to PyPI
In this lesson you will learn how to publish your package to TestPyPI
using [Hatch](https://hatch.pypa.io/latest/). There are 4 things that
you need to do to publish your Python package:
to PyPI. You need to:
to TestPyPI. You need to:

1. **Create a package development environment**
1. [**Build your package using `hatch build`**](../package-structure-code/python-package-distribution-files-sdist-wheel). Building a package is the process of turning your code into two types of distribution files: sdist and wheel. The wheel distribution file is particularly important for users who will `pip install` your package.
1. **Create an account on (test) PyPI**: You will need to create a PyPI account and associated token which provides permissions for you to upload your package.
1. **Publish to PyPI using `hatch publish`**
1. **Create an account on TestPyPI (or PyPI)**: You will need to create a TestPyPI account and associated token which provides permissions for you to upload your package. When you later publish your package to PyPI, you will need a separate PyPI account and token.
1. **Publish to TestPyPI using `hatch publish`**

In a future lesson, you will learn how to create an automated
GitHub action workflow that publishes an updated
Expand All @@ -79,7 +65,7 @@ version of your package to PyPI every time you create a GitHub release.
:class: tip

- [Learn more about what building a Python package is](../package-structure-code/python-package-distribution-files-sdist-wheel)
- [Learn more about package distribution file that PyPI needs called the wheel](#python-wheel)
- [Learn more about the package distribution file that PyPI needs called the wheel](#python-wheel)
- [Learn more about the package distribution file that conda-forge will need on PyPI called the sdist (source distribution)](#python-source-distribution)

:::
Expand All @@ -91,7 +77,7 @@ The first step in building your package is to create a development environment.
Use Hatch to create your environment.

```bash
# This will create a default envt with your package installed in editable mode
# This will create a default environment with your package installed in editable mode
> hatch env create
# If you have already created an environment this command will return Environment `default` already exists
```
Expand Down Expand Up @@ -200,23 +186,22 @@ dist/pyospackage-0.1.0-py3-none-any.whl
### <i class="fa-solid fa-wand-magic-sparkles"></i> Congratulations - you've created your Python package distribution files <i class="fa-solid fa-wand-magic-sparkles"></i>

You've now built your Python package and created your package distribution files. The next step is to setup
your account on testPyPI so you can publish your package.
your account on TestPyPI so you can publish your package.

## Step 3. Setup your test PyPI account
## Step 3. Setup your TestPyPI account

Next, you'll setup an account on Test PyPI. Remember that you
are using test PyPI here instead of the PyPI as a way to
safely learn how to publish a package without stressing the
real PyPI's servers.
Next, you'll setup an account on TestPyPI. Remember that you
are using TestPyPI here instead of the real PyPI as a way to
safely learn how to publish a package without accidentally "releasing" your package before it's ready.

:::{admonition} Test PyPI vs. PyPI
If you have a package that you are confident belongs on PyPI, all of the steps below will also work for you. When you publish using Hatch, you will call `hatch publish` to publish directly to PyPI instead of `hatch publish -r test` which publishes to Test PyPI.
:::{admonition} TestPyPI vs. PyPI
If you have a package that you are confident belongs on PyPI, all of the steps below will also work for you. When you publish using Hatch, you will call `hatch publish` to publish directly to PyPI instead of `hatch publish -r test` which publishes to TestPyPI.
:::

1. [Open up a web browser and go to the test PyPI website](https://test.pypi.org/).
1. [Open up a web browser and go to the TestPyPI website](https://test.pypi.org/).
2. [Create an account](https://test.pypi.org/account/register/) if you don't already have one. Be sure to store your password in a safe place!
3. Once you have an account setup, login to it.
4. Search on [https://test.pypi.org/](https://test.pypi.org/) to ensure that the package name that you have selected doesn't already exist. If you are using our test pyosPackage, then we suggest that you add your name or GitHub username to the end of the package name to ensure it's unique.
4. Search on [https://test.pypi.org/](https://test.pypi.org/) (and also on [https://pypi.org/](https://pypi.org/)) to ensure that the package name that you have selected doesn't already exist. If you are using our test pyosPackage, then we suggest that you add your name or GitHub username to the end of the package name to ensure it's unique.

Example: `pyosPackage_yourNameHere`.

Expand All @@ -228,10 +213,10 @@ Show them how to do this
:::

:::{figure-md} test-pypi-search
<img src="../images/tutorials/testpypi-search.png" alt="This is a screenshot of the test PyPI website. At the top in the search bar, you can see the search for pyosPackage. The search return says there were no results for pyosPackage Did you mean probpackage" width="700px">
<img src="../images/tutorials/testpypi-search.png" alt="This is a screenshot of the TestPyPI website. At the top in the search bar, you can see the search for pyosPackage. The search return says there were no results for pyosPackage Did you mean probpackage" width="700px">

Before you try to upload to test PyPI, check to see if the name of your package is already taken. You can do that using
the search box at the top of the test PyPI website.
Before you try to upload to TestPyPI, check to see if the name of your package is already taken. You can do that using
the search box at the top of the TestPyPI website.
:::

:::{admonition} Setup 2-factor (2FA) authentication
Expand All @@ -247,22 +232,19 @@ as of 1 January 2024.

## Step 4. Create a package upload token

To upload your package to PyPI, you will need to create a token. Ideally
this token is specific to the package that you are publishing.

However, if your package isn’t already on PyPI, then you will need to create a token for your account first and then create a package-specific token.
To upload your package to TestPyPI (or PyPI), you will need to create a token for your account first, and should then create a package-specific token. (If you completed this step previously, you can reuse the tokens when you upload your package again.)

:::{admonition} Why create package-specific tokens?

It's ideal to create a package-specific token. When you create an account wide token this allows anyone with access to then access all of your PyPI projects. By creating a package specific token, you are limiting the scope of the token to only your specific package. This is just a safe way to set things up for you particularly if you are collaborating with others on package development.
It's ideal to create a package-specific token. When you create an account-wide token this allows anyone with access to the account to then access all of your TestPyPI (or PyPI) projects. By creating a package-specific token, you are limiting the scope of the token to only your specific package. This is just a safe way to set things up for you particularly if you are collaborating with others on package development.
:::

### Follow the steps below to create your token.

- Login to test PyPI and go to your account settings
- Login to TestPyPI and go to your account settings
- Scroll down to the **API tokens** section
- Click on the **Add API Token** button
- If you are new to using PyPI and don't have any packages there yet, OR if you have other packages on PyPI but are uploading a new package, you will need to create an account-wide token.
- If you are new to using TestPyPI and don't have any packages there yet, OR if you have other packages on TestPyPI but are uploading a new package, you will need to create an account-wide token.
- When you create your token, be sure to copy the token value and store it in a secure place before closing that browser.

Your token should look something like this:
Expand All @@ -271,62 +253,62 @@ Your token should look something like this:

It should start with `pypi` followed by a dash and a bunch of characters.

### Upload to PyPI using Hatch
### Upload to TestPyPI using Hatch

Once you have your token, you are ready to publish to
PyPI.
TestPyPI.

- Run `hatch publish -r test`

`-r` stands for repository. In this case because you are publishing to test-PyPI you will use `-r test`. Hatch will then ask for a username and credentials.
`-r` stands for repository. In this case because you are publishing to TestPyPI you will use `-r test`. Hatch will then ask for a username and credentials.

- Add the word `__token__` for your username. This tells Test PyPI that you are using a token value rather than a username.
- Paste your PyPI token value in at the `Enter your credentials` prompt:
- Add the word `__token__` for your username. This tells TestPyPI that you are using a token value rather than a username.
- Paste your TestPyPI token value in at the `Enter your credentials` prompt:

```bash
❯ hatch publish -r test
Enter your username: __token__
Enter your credentials: <past-your-token-value-here>
Enter your credentials: <paste-your-token-value-here>
dist/pyospackage-0.1.0-py3-none-any.whl ... already exists
dist/pyospackage-0.1.0.tar.gz ... already exists

```

If your credentials are valid, and you have already run `hatch build`
and thus have your 2 distribution files in a `dist/` directory then
Hatch will publish your package to test PyPI.
Hatch will publish your package to TestPyPI.

Hatch also has a caching system so once you enter your credentials it
will remember them.

## Install your package from test PyPI
## Install your package from TestPyPI

Once your package upload is complete, you can install it from
test PYPI. You can find the installation instructions on the test PyPI
TestPYPI. You can find the installation instructions on the TestPyPI
landing page for your newly uploaded package.

:::{figure-md} testpypi-landing-page
<img src="../images/tutorials/test-pypi-package.png" alt="A screenshot of the test PyPI page for pyosPackage. It says pyosPackage 0.1.0 at the top with the pip install instructions below. The landing page of the package has information from the package's README file. " width="700px">
<img src="../images/tutorials/test-pypi-package.png" alt="A screenshot of the TestPyPI page for pyosPackage. It says pyosPackage 0.1.0 at the top with the pip install instructions below. The landing page of the package has information from the package's README file. " width="700px">

This is an example landing page for the pyosPackage that was just uploaded. Notice at the top of the page there is instruction for how to install the package from test PyPI. You can simply copy that code and use it to install your package from testPyPi locally.
This is an example landing page for the pyosPackage that was just uploaded. Notice at the top of the page there are instructions for how to install the package from TestPyPI. You can simply copy that code and use it to install your package from TestPyPI locally.
:::

As an example, [check out our pyOpenSci pyosPackage landing page on test PyPI](https://test.pypi.org/project/pyosPackage/). Notice that
As an example, [check out our pyOpenSci pyosPackage landing page on TestPyPI](https://test.pypi.org/project/pyosPackage/). Notice that
the page has information about the current package version and also
installation instructions as follows:

`pip install -i https://test.pypi.org/simple/ pyosPackage`

:::{important} Publishing to test.PyPI.org vs PyPI.org
While you can install from test PyPI it's not recommended that you publish to
testPyPI as a permanent way to install your package. Test PyPi is a perfect place to learn how to publish your package. But your end goal should be to publish to PyPI.org once you have figured out your workflow.
:::{important} Publishing to TestPyPI vs PyPI
While you can install from TestPyPI it's not recommended that you publish to
TestPyPI as a permanent way to install your package. In fact, you cannot, because TestPyPI may delete accounts after a time. TestPyPI is a perfect place to learn how to publish your package and test the installation process. But your end goal should be to publish to PyPI once you have figured out your workflow and your package is ready to deploy.
:::

### Time to install your package

- On your computer, activate the development environment that
you wish to install your newly published package in.
- Run the installation instructions for your package from test PyPI.
- Run the installation instructions for your package from TestPyPI.

::::{tab-set}

Expand All @@ -340,7 +322,7 @@ testPyPI as a permanent way to install your package. Test PyPi is a perfect plac

:::

:::{tab-item} venv mac / Linux
:::{tab-item} venv Mac / Linux

```bash
> hatch shell
Expand All @@ -351,7 +333,7 @@ testPyPI as a permanent way to install your package. Test PyPi is a perfect plac


:::{admonition} The value of end-to-end tools like hatch, flit and poetry
In this lesson you are using Hatch and hatchling to create, build and publish your Python Package. [Click here to learn about other packaging tools in the ecosystem.](../package-structure-code/python-package-build-tools.md)
In this lesson you are using Hatch and hatchling to create, build and publish your Python package. [Click here to learn about other packaging tools in the ecosystem.](../package-structure-code/python-package-build-tools.md)
:::

:::{todo}
Expand All @@ -378,7 +360,7 @@ If you plan to use your token regularly to publish to PyPI, we strongly recommen
a token specific to your new package.

To do this:
1. Go to test PyPI
1. Go to TestPyPI.
1. Navigate to the "Your Projects" section of your account
2. Click on the manage button for the project that you wish to add a token for
3. Go to settings
Expand All @@ -387,11 +369,11 @@ To do this:

And you're all done!

## You have published your package to (test) PyPI!
## You have published your package to TestPyPI!

Congratulations. You have now successfully published your package to test PyPI. If you have a package that is ready for real-world use on the real PyPI, then you can follow the same steps to publish it on PyPI.org.
Congratulations. You have now successfully published your package to TestPyPI. If you have a package that is ready for real-world use on the real PyPI, then you can follow the same steps (with the differences noted above) to publish it on PyPI.

Once you publish on PyPI.org, you can then easily add your package to the conda-forge ecosystem using the [grayskull](https://conda-forge.org/blog/posts/2020-03-05-grayskull/) tool.
Once you publish on PyPI, you can then easily add your package to the conda-forge ecosystem using the [grayskull](https://conda-forge.org/blog/posts/2020-03-05-grayskull/) tool.

You will learn how to do that in the next lesson.

Expand Down
Loading