I’ve been complaining about the Poetry project a lot, in particular about its use (or more precisely, the use of poetry-core) as a build system. In fact, it pretty much became a synonym of a footgun for me — and whenever I’m about to package some project using poetry-core, or switching to it, I’ve learned to expect some predictable mistake. I suppose the time has come to note all these pitfalls in a single blog post.
The nightmarish caret operator
One of the first things Poetry teaches us is to pin dependencies, SemVer-style. Well, I’m not complaining. I suppose it’s a reasonable compromise between pinning exact versions (which just asks for dependency conflicts between different packages), and leaving user at the mercy of breaking changes in dependencies. The problem is, Poetry teaches us to treat these pins in a wholesale, one-size-fits-all manner.
What I’m talking about is the (in)famous caret operator. I mean, I suppose it’s quite convenient for the general case of semantic versioning, where e.g. ^1.2.3 is handy short for >=1.2.3,<2.0.0, and works quite well for the non-exactly-SemVer case of ^0.2.3 for >=0.2.3,<0.3.0. However, the way it is presented as a panacea means that most of the time people use it for all their dependencies, whether it is meaningful there or not.
So some pins are correct, some are too strict and others are too lax. In the end, you get the worst of both worlds: you annoy distro packagers like us who have to keep relaxing your dependencies, and you don’t help users who still get incidental breakage. Some people even use the caret operator for packages that clearly don’t fit it at all. My favorite example is the equivalent of the following dependency:
tzdata = "^2023.3"
This actually suffers from two problems. Firstly, this package clearly uses CalVer rather than SemVer, so pinning to 2023 seems fishy. Secondly, since we are talking about timezone data, there is really no point in pinning at all — on the contrary, you always want to use up-to-date timezone data.
The misleading include key
When people want to control which files are included in the source distributions, they resort to the include and exclude keys. And they add “obvious” blocks like the following:
include = [
"CHANGELOG",
"README.md",
"LICENSE",
]
Except that this is entirely wrong! A plain entry in the include key is included both in source and in binary distribution. Or, to put it more clearly, this code causes the following files to be installed:
/usr/lib/python3.12/site-packages/CHANGELOG /usr/lib/python3.12/site-packages/LICENSE /usr/lib/python3.12/site-packages/README.md
What you need to do instead is to annotate every file with the desired format, i.e.:
include = [
{ path = "CHANGELOG", format = "sdist" },
{ path = "README.md", format = "sdist" },
{ path = "LICENSE", format = "sdist" },
]
Yes, this is absolutely confusing and counterintuitive. On top of that, even today the first example in the linked documentation is clearly wrong. And people keep repeating this mistake over and over again — I know because I keep sending pull requests fixing them, and there is no end to them! In fact, I’ve even seen people adding additional entries without the format just below entries that did have it!
Schrödinger’s optional dependency
Poetry has a custom way of declaring optional dependencies. You declare them just like a regular dependency, and add an optional key to it, e.g.:
[tool.poetry.dependencies]
python = "^3.7"
filetype = "^1.0.7"
deprecation = "^2.1.0"
# yaml-plugin extra
"ruamel.yaml" = {version = "^0.16.12", optional = true}
Well, so that last dependency is optional, right? Well, not necessarily! It is not, unless you actually add it to some dependency group, such as:
[tool.poetry.extras] yaml-plugin = ["ruamel.yaml"]
And again, this weird behavior leads to real problems. If you declare a dependency as optional, but forget to add it to some group, Poetry will just silently treat it as a required dependency. And this is really easy to miss, unless you actually look at the generated wheel metadata. A bug about confusing handling of optional dependencies has been filed back in 2020.
Summary
These are the handful of common issues I’ve repeatedly seen happening when people tried to use poetry-core as a build system. Sure, other PEP 517 backends aren’t perfect and have their own issues. For one, setuptools pretty much consists of tons of legacy, buggy code, deprecated bits everyone uses anyway, and is barely kept alive these days. People also fall into pitfalls there.
However, I have never seen any other Python or non-Python build system that would be as counterintuitive and mistake-prone as Poetry is. On top of that, implementing PEP 621 (the standard for pyproject.toml pretty much every other PEP 517 backend follows) took 3 years — and even today, Poetry still defaults to their own, nonstandard configuration format.
Whenever I criticize Poetry, people ask me about the alternatives. For completeness, let me repeat my PEP517 backend recommendations here:
For pure Python packages: use either flit-core (lightweight, simple, no dependencies), or hatchling (popular and quite powerful, and we have to deal with its disadvantages anyway). For Python packages with C extensions, meson-python combines the power and correctness of Meson with good Python integration. For Python packages with Rust extensions, Maturin is the way to go.