I’ve written previously on Python Packaging and in that article included details of how to automate publishing to PyPI from GitHub. This article details how to automatically publish your package to PyPI from GitLab.
Repository Configuration
CI Variables
The environment variables $TWINE_USERNAME (__token__) and $TWINE_PASSWORD which will be the token you generate for publishing on PyPI or Test PyPI. These are saved under the repository _Settings > CI/CD > Varialbes_ section and how to create and save these is described below.
CI Configuration
CI
GitLabs CI/CD is configured via a YAML file .gitlab-ci.yaml in the root of your project folder, a useful reference for writing these files is the .gitlab-ci.yml reference.
An example file from the tcx2gpx package is shown below (see here).
This defines the following…
image- the use of a Docker Python 3.11 image for running the pipeline.variables- Configures pre-commit to run and automatically fix issues found on pull requests.stages- the subsequent stages to run (NB thedebugstage which prints the environment variables is commented out).pylint- runs linting on Python 3.10 and 3.11.pytest- Runs tests on Python 3.10 and 3.11.pages- Builds the documentation pages.pypi- Builds and uploads the package to PyPI if the commit has a tag associated.
image: python:3.11
variables: # since we're not using merge request pipelines in this example, # we will configure the pre-commit job to
run on branch pipelines only. # If you ARE using merge request pipelines, you can omit this section
PRE_COMMIT_AUTO_FIX: '1' PRE_COMMIT_DEDUPLICATE_MR_AND_BRANCH: 'false' PRE_COMMIT_AUTO_FIX_BRANCH_ONLY: 'false'
before_script:
- python --version
- pip install .
# pre-commit autofix (https://gitlab.com/yesolutions/gitlab-ci-templates /
# https://stackoverflow.com/collectives/gitlab/articles/71270196/)
include: remote: https://gitlab.com/yesolutions/gitlab-ci-templates/raw/main/templates/pre-commit-autofix.yaml
stages: # - debug - pylint - pytest - pages - pypi
# print-all-env-vars-job:
# stage: debug
# script:
# - echo "GitLab CI/CD | Print all environment variables"
# - env
.pylint: script:
- pip install pylint pytest
- pylint --rcfile .pylintrc tcx2gpx/
- pylint --rcfile .pylintrc tests/
pylint-3-10: extends: .pylint stage: pylint image: python:3.10 allow_failure: true
pylint-3-11: extends: .pylint stage: pylint image: python:3.11 allow_failure: true
.pytest: script:
- pip install pytest pytest-cov
- python -m "pytest"
pytest-3-10: extends: .pytest stage: pytest image: python:3.10 allow_failure: true
pytest-3-11: extends: .pytest stage: pytest image: python:3.11 coverage: /(?i)total.*?
(100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/
pages: stage: pages rules:
- if: $CI_COMMIT_BRANCH == "master" script:
- pip install .[docs]
- cd docs
- git fetch --tags
- git tag -l
- make html
- mkdir ../public
- mv _build/html/* ../public/ artifacts: paths:
- public
pypi: stage: pypi rules:
- if: $CI_COMMIT_TAG script:
- pip install .[pypi]
- pip install build
- python -m build
- twine upload --non-interactive --repository pypi dist/*The pypi stage is named and a rule is defined that says to only run this stage if the value of the environment variable $CI_COMMIT_TAG is True. This only happens when a commit has a (protected 😉) tag.
The script section then installs the package along with the project.optional-dependencies defined in the pypi section of the pyproject.toml.
The package is then built using build and twine is used to push the to push the built package to PyPI.
PyPI Tokens
You should first test building and deploying to the Test PyPI and when this is working simply switch to using the main PyPI. To do so you will need to create an account on both1. Once you have set yourself up with an account you can generate an API token to authenticate with PyPI. After verifying your email got to Account Settings and select Add API token. These are generated once so copy and paste it into the .pypirc of your project (add this file to your .gitignore so it doesn’t accidentally get added). Remember to do this twice, once for PyPI and once for Test PyPI and once for PyPI for reference.
[testpypi]
username = __token__
password = pypi-<token_value>
[pypi]
username = __token__
password = pypi-<token_value>In GitLab go to your repositories Settings > CI/CD > Variables and add two new variables TWINE_USERNAME with the value __token__ and TWINE_PASSWORD with the token for your account on Test PyPI (remember it should include the prefix pypi- as shown in the above example .pypirc). You have options on how these variables are used and should ensure that all three check boxes are selected, this enables…
- Protect variable Export variable to pipelines running on protected branches and tags only.
- Mask variable Mask this variable in job logs if it meets regular expression requirements.
- Expand variable reference
$will be treated as the start of a reference to another variable.
Testing
Now that you are setup you can test your configuration. To do so you need to first use the API key from the Test PyPI server that you created as the value for $TWINE_PASSWORD (see above) and set the repository twine --repository option to testpypi. Your pypi stage should look like the following…
pypi:
stage: pypi
rules:
- if: $CI_COMMIT_TAG
script:
- pip install .[pypi]
- pip install build
- python -m build
- twine upload --non-interactive --repository testpypi dist/*Once this is set create a tag for the current commit using the Code > Tags settings from the left menu of your repository and then the New tag button on the top right. The tag you create should match the wild card pattern you have set for protecting tags and it should comply to the Public version identifiers specified in PEP440. On creation it triggers the Pipeline, you can check progress and status by navigating to CI/CD > Pipelines and then viewing it. The pypi job should complete and you should be able to navigate to your package on Test PyPI. You can find it under your account settings.
If you find there is a problem you will have to correct it and either delete the tag you created and try again or increment the version. PyPI, and in turn Test PyPI which is a mirror with the same functionality, does not permit uploading packages with a version number that already exists.
Publishing to PyPI
Once you have successfully published to the Test PyPI you are ready to publish to PyPI. There three things you need to do.
- Delete the existing tag, if you want to apply the same tag to publish to PyPI you can do so.
- Modify the repository option to point to PyPI
--repository pypi(or remove it, the default is PyPI). - Change the key stored in the
$TWINE_PASSWORDto that which you generated for PyPI instead of the one used for testing with Test PyPI.
Once you have done so you can create a new tag and the upload will be made to PyPI.
Releases
An alternative way to apply tags to commits is to make a Releases. In creating a release you apply a tag to the current commit. In addition GitLab will build and compress snapshot of the files and you can add Release Notes detailing what has changed. GitLab will automatically build release artifacts of your repository and make them available for download directly from GitLab.
Links
Python Packaging
GitLab Documentation
Footnotes
PyPI now enforces Two Factor Authentication (2FA) for new accounts, see 2FA Enforcement for New User Registrations↩︎
Reuse
Citation
@online{shephard2023,
author = {Neil Shephard},
title = {GitLab {CI} - {Automatic} {Publishing} to {PyPI}},
date = {2023-10-03},
url = {https://blog.nshephard.dev//posts/gitlab-ci-pypi},
langid = {en}
}