Python Poetry: A Poem About Dependency Management in Python

Software Engineer at ESK Solutions.

Have you ever started a new Python project or revisited an old one, and while setting up the environment, you get that annoying message "cannot install dependency X because version of package has conflicting dependencies" with the project dependencies? Then you check the requirements file and find that the offending dependency doesn't have a pinned version, or worse, it's not there at all. You are not alone; it's happened to me once or twice.

When this happens, the best solution is always:

1. Check the version from a colleague.
2. Add it to the requirements file.
3. Re-run the installation process.
4. Hope it doesn't happen with the next dependency.
5. Repeat.

You see, this is a very reactive process because it doesn't prevent the situation from recurring, and sooner or later it will. That thought bothered me. Is there a solution to this common problem? Something like npm for front-end projects?

The Old Way

The usual way to install dependencies in a Python project is using pip. With pip, you add a dependency by running pip install X, then write it with a version pin (e.g., requests==2.26.0) in a requirements.txt file, so that the next time a newcomer just needs to run pip install -r requirements.txt to install them.

Python Poetry CLI with commands, TOML, and lock files.

The problem with this approach is that if dependencies have sub-dependencies, you may not always get the same version because they aren't listed in the requirements.txt file. Another problem with this process is that you might forget to add a dependency to the requirements file. That's where Python Poetry comes in.

What Makes Python Poetry Attractive

Poetry uses a TOML file instead of a text file to track dependencies, along with a lock file to track dependencies and sub-dependencies. Each time you install project dependencies, you will always get the same versions as in the lock file. Poetry goes further by storing the package hash, as some maintainers might update a package without changing its version.

Poetry comes with a convenient CLI for installing dependencies, so pip is no longer needed. When installing dependencies, they are automatically added to the TOML file, saving you from having to remember the packages you add to the project.

Python Poetry enforces the use of virtual environments. If you aren't already using a virtual environment, it will create a new one to isolate dependencies between different projects. If you are using Docker, you can disable this feature since there is no point in using virtual environments inside containers.

Unlike the old way, Poetry allows you to update all packages, and depending on the dependency specification you use, it can be a patch or minor update (for major updates, thorough research is recommended).

A dependency specification is a version constraint you apply to your dependencies. You can use caret, tilde, or wildcard constraints, or even specify a range of acceptable versions. You can also use git, path, and URL dependencies, as well as source dependencies.

Another nice feature of Poetry is that when you remove a dependency you no longer need, its sub-dependencies are also removed, which is not the case with pip. This avoids orphan dependencies in your environment.

Finally, Poetry allows you to separate main dependencies needed for the project to run from development dependencies used mainly for debugging. This allows for a production environment free of unnecessary dependencies.

It's worth noting that Poetry can also package and publish libraries, but I will focus only on dependency management.

Composing Your First Poem

There is no better way to understand something than to see it for yourself, so we'll run a short guide on migrating an existing project to Python Poetry. I will assume you have Python 3.7+ and git installed. If you feel bold, you can use a project you have or are working on, but I will provide a Django project mockup for this tutorial. The version of Python Poetry at the time of writing is 1.3.1.

A Python developer puzzled by a requirements.txt file.

First of all, you can install Python Poetry by running the command:

curl -sSL https://install.python-poetry.org | python3-

You can verify that Poetry is successfully installed by running poetry --version. You can also run poetry list to display a list of available commands.

Now that the script is ready, let's proceed with the migration.

Initializing Poetry

The first step is to initialize Python Poetry in the project, so run the command poetry init inside the repository. An interactive prompt will appear to set up the TOML file. You can use the default suggestions, but make sure your Python version matches the "Compatible Python version" field. When you get to the section "Do you want to define your main dependencies interactively?" simply type "no". Then it will ask if you want to define dev dependencies interactively; again type "no". Finally, it will ask for confirmation of the settings; enter "yes".

Migrating Dependencies

Since you already have a requirements.txt file, the easiest way to migrate dependencies is to run the command:

cat requirements.txt | xargs poetry add

Poetry will add each dependency to the TOML file and attempt to resolve any conflicts, then write the lock file. If you are not using a virtual environment, the dependencies will be installed in a newly created virtual environment.

Now that Poetry has all your dependencies in its list, you can check them by running poetry show.

Since the requirements.txt file is no longer needed, you can get rid of it.

Cleaning Up Nested Dependencies

Since you likely have sub-dependencies listed as main ones, we can get rid of them. When you want to update top-level dependencies, there will be no constraints.

Run the command:

poetry show --tree

And you'll see which ones are sub-dependencies.

Creating Updatable Dependencies

The last step is to make dependencies updatable. Personally, I only use patch updates because they usually contain fixes and security patches. Minor and major updates I only do after reading the changelogs and being sure they are what we want to add to the project.

You can use the dependency specification (mentioned earlier in the article) that suits you best. You can reopen the TOML file and edit the version of each dependency by adding a version constraint (tilde, caret, wildcard, etc.). In my case, I used a tilde constraint.

Now the Django and djangorestframework dependencies can receive updates (patches). The next time an update is released for any of these dependencies, you can update them by running the command:

poetry update

This will attempt to update all dependencies. You can also specify which dependencies to update. If updates are found, Poetry will resolve dependency conflicts and run the update. After that, it will update the lock file.

In this case, if you run the update and suppose you make the same changes as I did, you'll see that djangorestframework updates from version 3.10.0 to 3.10.3, which is the latest patch at the time of writing.

A diverse team of Python developers working on a computer screen.

Next Steps

From now on, you can use Poetry to add and remove dependencies from your project using only the CLI. If you make an update outside of the constraints, you'll need to run the add command with the new version.

For newcomers to the project, to get exactly the same package versions as the last person who updated the lock file, just run poetry install.

Summary

Like npm, Python Poetry is a powerful tool for dependency management that solves many problems that arise when using a requirements.txt file with pip. It also offers more functionality, and the CLI is very easy to use. It can be applied to both new and existing Python projects, making it easy to add, remove, and update dependencies as the project evolves.