You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Poetry and CI builds -- best practices, knowledge share (travisci), and discussion #366
I have searched the issues of this repo and believe that this is not a duplicate.
I have searched the documentation and believe that my question is not covered.
Question
Hi! I've been working with poetry recently and am happy that I can publish packages to pypi or my own hosted package repository with it. I would like to share my current setup with TravisCI in hopes that it could help others. I would also like to learn from those who have more experience with poetry to get tips on how I might make this setup better, or see how it works with other continuous integration platforms, or other more in depth CI/CD requirements.
My requirements for travis-ci are fairly common.
Use poetry to install my project dependencies and dev dependencies.
Run lint and unit tests for my project.
When the project is git tagged, to build and publish my project to my companies internal pypi registry.
I have in the past used the built in travis pypi deployment, but it requires a setup.py ( which I don't have anymore! 🙌). So instead I'm running my poetry publish as a script deployment when I tag my repo.
So when master is at a spot where I want to deploy a new version of the package. I do something like.
poetry version minor
git commit -am 'bumped the version'
git tag <version># SIDE NOTE: it would be nice to be able to do `git tag $(poetry version --show)`# or possibly have the bump command output the NEW_VERSION=poetry version minor --output
git push --tags
In order to configure poetry with the credentials to push to our repository I have set $MYPYPI_USER and $MYPYPI_PASSencrypted environment variables in travis.
That's what I have. Cheers 🍺
Reacted by Jāzeps Baško, Nicolas Hainaux, Yinwei (Charlie) Zhang, Daniel Knell, Anton Kachurin, Bruno Oliveira, Andrii Oriekhov, Tim, Mantas Zimnickas, zhouquan and 34 more
language: pythonpython: "3.6"dist: xenialstages:
- lint
- test
- name: deployif: tag IS presentcache:
pip: truedirectories:
- "$HOME/.cache/pre-commit"jobs:
include:
- stage: lintinstall:
- pip install pre-commit
- pre-commit install-hooksscript:
- pre-commit run --all-files
- stage: testinstall:
- pip install --upgrade pip
- pip install poetry
- poetry install -vscript:
- pytest --cov
- stage: deployscript:
- echo Deploying to PyPI...before_deploy:
# User and password environment variables are set as hidden variables through# the web interface in the project settings.
- pip install --upgrade pip
- pip install poetry
- poetry config http-basic.pypi $PYPI_USER $PYPI_PASS
- poetry builddeploy:
provider: scriptscript: poetry publishskip_cleanup: trueon:
all_branches: true # Travis recognizes tag names as "branches"condition: $TRAVIS_BUILD_STAGE_NAME = Deployrepo: daltonmaag/statmaketags: true
Notes:
I use pre-commit to lint my sources, which I put into its own stage. The lint stage is run before test and deploy is run.
Pip packages and the pre-commit installation are cached to speed up stuff.
The deploy stage is only enabled if Travis is looking at a tagged commit. It's more of a hack, so the deploy: section at the end can test if it's run from the deploy stage (see the condition). Reading the output of the stages makes me think it would otherwise deploy after every stage, but I'm not sure. That would be a gross misfeature.
Reacted by Will Vaughn, drunkwcodes, Evgeny Kemerov, Sebastián Ramírez, Julian and Yegor Statkevich
@FranklinYu isn't that true of all dev deps, say pytest? To test a package I need pytest, so I typically install all dev deps with poetry install which may create conflicts or, more often for me, hide a missing dep. I had the same problem before using poetry, just didn't think about it.
@piccolbo Bad things happen if you install poetry via pip but have it configured not to create a virtual environment. (As far as I can tell this isn't relevant to the configs discussed above though.)
It might be possible to speed up your builds if you don't have poetry create a virtualenvironment, and rely on the installer script to ensure the vendorized dependencies are present. (This probably depends on your CI system to some degree though.)
@piccolbo I'm lost at what you mean. If you add pytest through Poetry then it shouldn't conflict with other dependencies. This is the whole point of Poetry, isn't it?
@FranklinYu I consider a conflict when even Poetry can not satisfy all dep constraints. Maybe I am not using the term in the most appropriate way. More generally, I was referring to the fact the dev environment is more complicated than the regular one but the only one in which we can test. So one may be unable to test the environment a user would get in say an empty env. Let's say mypackage requires pandas >=3.14.15 but installing pytest requires pandas >=4.0.0 (I am making this up). Can't test with pandas <4 anymore. Maybe I am worrying for no reason since one can't test all dep combinations anyway.
Let's say mypackage requires pandas >=3.14.15 but installing pytest requires pandas >=4.0.0 (I am making this up).
In this case Poetry will try to come up with a solution (like finding another pytest version). When this is impossible, it would raise an error before trying to install anything. This happens in version resolution, not during installation. This is the point of using Poetry instead of requirements.txt.
@FranklinYu@dmontagu thanks for your answers but they address different questions. Unless I find a better way to explain my question, I guess it's better to leave it at that.
# Global --------------------------variables:
PIP_CACHE_DIR: "${CI_PROJECT_DIR}/.cache/pip"cache:
key: "${CI_JOB_NAME}"paths:
- .cache/pip
- .venvstages:
- quality
- tests# Jobs templates ------------------.install-deps-template: &install-depsbefore_script:
- pip install poetry
- poetry --version
- poetry config settings.virtualenvs.in-project true
- poetry install -vv.quality-template: &quality<<: *install-depsimage: python:3.6stage: quality.test-template: &test<<: *install-depsstage: testscoverage: '/TOTAL.*\s(\d+\.\d+\%)/'script: make testartifacts:
paths:
- tests/logswhen: alwaysexpire_in: 1 week# Quality jobs ----------------------check-bandit:
<<: *qualityscript: make check-banditcheck-black:
<<: *qualityscript: make check-blackcheck-flake8:
<<: *qualityscript: make check-flake8check-isort:
<<: *qualityscript: make check-isortcheck-safety:
<<: *qualityscript: make check-safety# Tests jobs ------------------------python3.6:
<<: *testimage: python:3.6python3.7:
<<: *testimage: python:3.7python3.8:
<<: *testimage: python:3.8
Reacted by Nikolaus Waxweiler, Doron Atuar, Takahiro Ueda, Yi Liu, Maciej Lapinski, Guilhem Saurel, Pavel Řežábek, Matt, Felipe Villalobos, Fellipe Souto and 17 more
Adding to the conversation here, I took some of the recommendations from this thread, as well as others, and mixed in some of my own past patterns, and came up with the following .travis.yml, found here:
@earonesty In case you're using Tox, this suggestion solved a keyring error I had been getting with poetry config in CI/CD: jaraco/keyring#283 (comment)
- name: Set up Poetry cache for Python dependenciesuses: actions/cache@v2if: startsWith(runner.os, 'Linux')with:
path: ~/.cache/pypoetrykey: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}restore-keys: ${{ runner.os }}-poetry-
Use the custom installer
Installing Poetry via pip can lead to dependency conflicts, so the custom installer is recommended. The command listed in the docs exits in GitHub Actions with 127 (not on $PATH).
There are some additional modifications needed for GitHub Actions:
Poetry allows config from the poetry config command, or by environment variables. Environment variables are a more dependable way to configure Poetry in CI.
Bonus: automated dependency updates with Dependabot
Dependabot now offers automated version updates, with (preliminary) support for Poetry 🎉. If you have access to the Dependabot beta, set up .github/dependabot.yml as described in the docs:
Dependabot will now send you PRs when dependency updates are available. Although package-ecosystem must be set to pip, it will pick up the pyproject.toml and poetry.lock. Check the status of the repo at Insights -> Dependency graph -> Dependabot.
Reacted by Timothée Mazzucotelli, Will Vaughn, Arnaud Rocher, Maurizio Tomasi, Valentin Berlier, GHadari, Mieszko Bańczerowski and Kyle King
Reacted by Timothée Mazzucotelli, Giulio Starace and Amit Gal
# Global --------------------------variables:
PIP_CACHE_DIR: "${CI_PROJECT_DIR}/.cache/pip"cache:
key: "${CI_JOB_NAME}"paths:
- .cache/pip
- .venvstages:
- quality
- tests# Jobs templates ------------------.install-deps-template: &install-depsbefore_script:
- pip install poetry
- poetry --version
- poetry config settings.virtualenvs.in-project true
- poetry install -vv.quality-template: &quality<<: *install-depsimage: python:3.6stage: quality.test-template: &test<<: *install-depsstage: testscoverage: '/TOTAL.*\s(\d+\.\d+\%)/'script: make testartifacts:
paths:
- tests/logswhen: alwaysexpire_in: 1 week# Quality jobs ----------------------check-bandit:
<<: *qualityscript: make check-banditcheck-black:
<<: *qualityscript: make check-blackcheck-flake8:
<<: *qualityscript: make check-flake8check-isort:
<<: *qualityscript: make check-isortcheck-safety:
<<: *qualityscript: make check-safety# Tests jobs ------------------------python3.6:
<<: *testimage: python:3.6python3.7:
<<: *testimage: python:3.7python3.8:
<<: *testimage: python:3.8
This is great. @pawamoy , do you have any pointers on what PIP_CACHE_DIR does exactly? It doesn't seem widely used. I am trying to get poetry to install into the local directory when doing pip install poetry . For this, the --target options works. It allows us to install the entire thing, including modules and bin/aries into a subdirectory of the working directory. This allows for caching.
However, it also means we have to set PATH as well as PYTHONPATH to find that new custom location. That is not the most sexy approach. PIP_CACHE_DIRseems to alleviate this, but it does not actually change the install path. So as far as I can see, your pip install poetry is not cached and runs each time, can you confirm this?
Thanks for your inspirational file!
Update 2020-11-12
I have split this into two steps for now:
variables:
PIP_DOWNLOAD_DIR: ".pip"before_script:
# Allow caching by only downloading first:
- pip download --dest=${PIP_DOWNLOAD_DIR} poetry # STEP 1
- pip install --find-links=${PIP_DOWNLOAD_DIR} poetry # STEP 2# Make available for caching by installing to current directory:
- poetry config virtualenvs.in-project true
- poetry install -vv
pip download will just download the dependencies, allowing them to be cached. A later pip install --find-links to that same directory allows to install into the system (container)-wide Python. This allows us to ignore all of PATH and PYTHONPATH, it will Just Work™. The downside is that the install process itself is not cached and will have to rerun.
@pawamoy Thanks a lot for sharing, I discovered you could have templates in .gitlab-ci.yml with your example. On the other hand, is there a reason to run quality check jobs in parallel instead of running them sequentially?
Since they have no side effects I feel like its a waste of ressources (ci/cd pipeline minutes).
@cglacet You mean the other way around, parallel v sequentially? Towards this, you might like the needs keyword. It allows you to build a DAG of job dependencies, speeding stuff up.
The templates used there are YAML anchors, so not GitLab-specific. The extends allows you to do the same thing (essentially have job templates) and is GitLab specific. I find it more readable, but don't know if features differ. For simple stuff, they do the same.
Lastly, if your config is Makefile-based and has calls make <target> in each job's script, where the job name corresponds to target names, those concepts can be combined:
.make:
# Note we cannot have `default: script:`, so this approach works better.script: make ${CI_JOB_NAME}
Then, later, for example:
preflight:
extends: .makestage: .pre
(The .pre stage is always available, as is .post, as well as build, test, deploy, if no other stages are defined). The above will call make preflight, with the term preflight only occurring once in the config. Pretty dry! I suppose it gets more cumbersome to cross-check for existing make targets though.
@alexpovel about PIP_CACHE_DIR, I'm not sure anymore 😅 I guess I just picked it from another example somewhere 🙂 Thanks for sharing your solution, this is interesting!
@cglacet I think what you mean is: why don't I run all the checks in a single job, to avoid installing deps 5 times instead of just one? Well indeed, it's a waste of resources. I think I just liked seeing more green checkmarks in GitLab CI. But if wasting time and resources by installing too many deps is an issue to you (well, it should be to everyone), you can always reduce the installed dependencies to a minimum by making use of extras! For example, a flake8 extra with flake8 and all its plugins, then a tests extra with only the dependencies required for running the tests, etc. You'd then adapt the "install deps" step to make use of these extras. This way you can have your parallel jobs without any waste 🙂 And it's faster than sequential jobs 😉
@alexpovel I didn't know all these GitLab CI configuration features, very nice! The snippet I shared was written when GitLab didn't have the extends ability yet 🙂 The trick with make and the CI job name is really nice, thanks for sharing!
you can always reduce the installed dependencies to a minimum by making use of extras ... This way you can have your parallel jobs without any waste
I feel it's still a waste because you have to start 5 docker instances instead of one. But I agree that having parallel jobs isn't only a cost, reducing user build time is clearly a plus if you build often and don't want to wait. I guess the choice depend on the use case. From my personal perspective I would love to remain under 400 minutes per month so even if it's only saving 20% I would gladly sacrifice a x3 speedup factor on the user build time.
Now I just need to wait and see when I change my mind about this 😄 . The future me will maybe think I'm an ignorant.
For those who wonder how to publish to gitlab from your CI, I struggled finding the info so here is how I did it (I'm not sure that's the right way, but it's working decently):
@alexpovel No I meant it this way. When building in parallel you save "real time" because you actually wait less for the whole build to finish. On the other hand consume more computation time (because in this case you start several containers and install the dependencies many times).
I am trying to publish a python wrapper for a C# library and hope to build a bunch of wheels for different OS using Github ci. I can build using poetry on each os separately in a different task, but when I hit publish only one set of wheels arrives. Perhaps anyone has a recipe with multiple artifacts published all at once? 🙏
@Casyfill I don't know which python package index you're publishing to (I'm assuming pypi.org? which I'm not familiar with) but it's likely it doesn't allow you to publish a package with the same name and version number as an existing one, even if it's a variant like a different OS or Python version.
Instead of building and publishing your packages one by one, the solution is to build everything as you do now, and then have a separate job that gathers the packages and publishes them in one command.
Activity
epage commentedon Oct 8, 2018
If someone is checking the lock file in, it'd also be nice to include best practices for updating it. Ideally we'd get pyup support for it.
madig commentedon Feb 27, 2019
Your
.travis.ymlinspired my own:Notes:
lintstage is run beforetestanddeployis run.deploy:section at the end can test if it's run from thedeploystage (see thecondition). Reading the output of the stages makes me think it would otherwise deploy after every stage, but I'm not sure. That would be a gross misfeature.vancouverwill commentedon May 2, 2019
on Gitlab setting
virtualenvs.in-projectto true worked for me and then caching the.venvdirectoryThe relevant section of the
.gitlab-ci.ymlfile arematthieugouel commentedon May 5, 2019
I came into a slightly different
.travis-ci.ymlbased on your suggestions :FranklinYu commentedon Jun 13, 2019
Is it recommended to install poetry through
pip? From README:In addition #166 may be related.
piccolbo commentedon Sep 18, 2019
@FranklinYu isn't that true of all dev deps, say pytest? To test a package I need pytest, so I typically install all dev deps with poetry install which may create conflicts or, more often for me, hide a missing dep. I had the same problem before using poetry, just didn't think about it.
dmontagu commentedon Sep 19, 2019
@piccolbo Bad things happen if you install poetry via pip but have it configured not to create a virtual environment. (As far as I can tell this isn't relevant to the configs discussed above though.)
It might be possible to speed up your builds if you don't have poetry create a virtualenvironment, and rely on the installer script to ensure the vendorized dependencies are present. (This probably depends on your CI system to some degree though.)
FranklinYu commentedon Sep 19, 2019
@piccolbo I'm lost at what you mean. If you add pytest through Poetry then it shouldn't conflict with other dependencies. This is the whole point of Poetry, isn't it?
piccolbo commentedon Sep 19, 2019
@FranklinYu I consider a conflict when even Poetry can not satisfy all dep constraints. Maybe I am not using the term in the most appropriate way. More generally, I was referring to the fact the dev environment is more complicated than the regular one but the only one in which we can test. So one may be unable to test the environment a user would get in say an empty env. Let's say mypackage requires pandas >=3.14.15 but installing pytest requires pandas >=4.0.0 (I am making this up). Can't test with pandas <4 anymore. Maybe I am worrying for no reason since one can't test all dep combinations anyway.
FranklinYu commentedon Sep 20, 2019
In this case Poetry will try to come up with a solution (like finding another pytest version). When this is impossible, it would raise an error before trying to install anything. This happens in version resolution, not during installation. This is the point of using Poetry instead of
requirements.txt.piccolbo commentedon Sep 20, 2019
@FranklinYu @dmontagu thanks for your answers but they address different questions. Unless I find a better way to explain my question, I guess it's better to leave it at that.
pawamoy commentedon Nov 29, 2019
Can I continue the sharing 🙂 ?
Here is my typical
.gitlab-ci.yml:[CI SKIP] Add workaround for travis releases
tarkatronic commentedon May 27, 2020
Adding to the conversation here, I took some of the recommendations from this thread, as well as others, and mixed in some of my own past patterns, and came up with the following
.travis.yml, found here:This has allowed me to continue using
toxfor development, with the following:4 remaining items
sisp commentedon Sep 2, 2020
@earonesty In case you're using Tox, this suggestion solved a keyring error I had been getting with
poetry configin CI/CD: jaraco/keyring#283 (comment)br3ndonland commentedon Sep 12, 2020
Poetry tips for GitHub Actions workflows
Use caching to speed up workflows
Use actions/cache with a variation on their
pipcache example to cache Poetry dependencies for faster installation.Use the custom installer
Installing Poetry via
pipcan lead to dependency conflicts, so the custom installer is recommended. The command listed in the docs exits in GitHub Actions with127(not on$PATH).There are some additional modifications needed for GitHub Actions:
-yto avoid prompts.$GITHUB_PATH(note that the::set-envsyntax has been deprecated).poetry installto separate step to ensure Poetry is on$GITHUB_PATH.Use environment variables for config
Poetry allows config from the
poetry configcommand, or by environment variables. Environment variables are a more dependable way to configure Poetry in CI.Build and publish in one step
${{ secrets.PYPI_TOKEN }}(secret name isPYPI_TOKENin this example, and username for PyPI tokens is__token__).poetry publish --buildto build and publish in one step.That's why they call it Poetry. Beautiful.
Example workflow
Expand this details element for an example workflow from br3ndonland/inboard that uses these tips.
Bonus: automated dependency updates with Dependabot
Dependabot now offers automated version updates, with (preliminary) support for Poetry 🎉. If you have access to the Dependabot beta, set up .github/dependabot.yml as described in the docs:
Dependabot will now send you PRs when dependency updates are available. Although
package-ecosystemmust be set topip, it will pick up the pyproject.toml and poetry.lock. Check the status of the repo at Insights -> Dependency graph -> Dependabot.Use Poetry custom installer in GitHub Actions
Use Poetry custom installer in GitHub Actions
alexpovel commentedon Nov 10, 2020
This is great. @pawamoy , do you have any pointers on what
PIP_CACHE_DIRdoes exactly? It doesn't seem widely used. I am trying to getpoetryto install into the local directory when doingpip install poetry. For this, the--targetoptions works. It allows us to install the entire thing, including modules andbin/aries into a subdirectory of the working directory. This allows for caching.However, it also means we have to set
PATHas well asPYTHONPATHto find that new custom location. That is not the most sexy approach.PIP_CACHE_DIRseems to alleviate this, but it does not actually change the install path. So as far as I can see, yourpip install poetryis not cached and runs each time, can you confirm this?Thanks for your inspirational file!
Update 2020-11-12
I have split this into two steps for now:
pip downloadwill just download the dependencies, allowing them to be cached. A laterpip install --find-linksto that same directory allows to install into the system (container)-wide Python. This allows us to ignore all ofPATHandPYTHONPATH, it will Just Work™. The downside is that the install process itself is not cached and will have to rerun.cglacet commentedon Nov 13, 2020
@pawamoy Thanks a lot for sharing, I discovered you could have templates in
.gitlab-ci.ymlwith your example. On the other hand, is there a reason to run quality check jobs in parallel instead of running them sequentially?Since they have no side effects I feel like its a waste of ressources (ci/cd pipeline minutes).
Thanks again for sharing your example.
alexpovel commentedon Nov 13, 2020
@cglacet You mean the other way around, parallel v sequentially? Towards this, you might like the
needskeyword. It allows you to build a DAG of job dependencies, speeding stuff up.The templates used there are YAML anchors, so not GitLab-specific. The
extendsallows you to do the same thing (essentially have job templates) and is GitLab specific. I find it more readable, but don't know if features differ. For simple stuff, they do the same.Lastly, if your config is
Makefile-based and has callsmake <target>in each job's script, where the job name corresponds to target names, those concepts can be combined:Then, later, for example:
(The
.prestage is always available, as is.post, as well asbuild,test,deploy, if no otherstagesare defined). The above will callmake preflight, with the termpreflightonly occurring once in the config. Pretty dry! I suppose it gets more cumbersome to cross-check for existingmaketargets though.pawamoy commentedon Nov 13, 2020
@alexpovel about
PIP_CACHE_DIR, I'm not sure anymore 😅 I guess I just picked it from another example somewhere 🙂 Thanks for sharing your solution, this is interesting!@cglacet I think what you mean is: why don't I run all the checks in a single job, to avoid installing deps 5 times instead of just one? Well indeed, it's a waste of resources. I think I just liked seeing more green checkmarks in GitLab CI. But if wasting time and resources by installing too many deps is an issue to you (well, it should be to everyone), you can always reduce the installed dependencies to a minimum by making use of
extras! For example, aflake8extra with flake8 and all its plugins, then atestsextra with only the dependencies required for running the tests, etc. You'd then adapt the "install deps" step to make use of these extras. This way you can have your parallel jobs without any waste 🙂 And it's faster than sequential jobs 😉@alexpovel I didn't know all these GitLab CI configuration features, very nice! The snippet I shared was written when GitLab didn't have the
extendsability yet 🙂 The trick with make and the CI job name is really nice, thanks for sharing!cglacet commentedon Nov 19, 2020
@pawamoy
I feel it's still a waste because you have to start 5 docker instances instead of one. But I agree that having parallel jobs isn't only a cost, reducing user build time is clearly a plus if you build often and don't want to wait. I guess the choice depend on the use case. From my personal perspective I would love to remain under 400 minutes per month so even if it's only saving 20% I would gladly sacrifice a x3 speedup factor on the user build time.
Now I just need to wait and see when I change my mind about this 😄 . The future me will maybe think I'm an ignorant.
For those who wonder how to publish to gitlab from your CI, I struggled finding the info so here is how I did it (I'm not sure that's the right way, but it's working decently):
In my Makefile:
A link that might help building better CI on gitlab: Predefined environment variables reference.
cglacet commentedon Nov 19, 2020
@alexpovel No I meant it this way. When building in parallel you save "real time" because you actually wait less for the whole build to finish. On the other hand consume more computation time (because in this case you start several containers and install the dependencies many times).
Casyfill commentedon Mar 28, 2021
Thanks for putting this together.
I am trying to publish a python wrapper for a C# library and hope to build a bunch of wheels for different OS using Github ci. I can build using poetry on each os separately in a different task, but when I hit
publishonly one set of wheels arrives. Perhaps anyone has a recipe with multiple artifacts published all at once? 🙏1ace commentedon Mar 29, 2021
@Casyfill I don't know which python package index you're publishing to (I'm assuming pypi.org? which I'm not familiar with) but it's likely it doesn't allow you to publish a package with the same name and version number as an existing one, even if it's a variant like a different OS or Python version.
Instead of building and publishing your packages one by one, the solution is to build everything as you do now, and then have a separate job that gathers the packages and publishes them in one command.
Casyfill commentedon Mar 30, 2021
Thanks, @1ace ! That's what I thought. It turned out to be easier than I thought, actually. Here is the link to our implementation:
https://github.com/Casyfill/pyCombo/blob/master/.github/workflows/release.yml