Managing Python Versions in 2024

Isolation and reproducibility

The two principles guiding my Python management are isolation and reproducibility. Isolation is the idea that you should be able to install and run a project without affecting other projects or your system. Reproducibility is the idea that you should be able to install and run a project on any machine without affecting the machine. In practice, this means that you should be able to install and run a project without affecting other projects or your system.

These principles are especially important in a professional setting where you may have multiple projects and multiple developers. Thus, the tools and approach I use to manage Python and Python packages are designed to isolate each project from each other and from the system, and to make it easy to reproduce the environment for each project.

Managing Python installations

First is to differentiate between managing Python and managing package versions. Most, if not all, systems will come with Python pre-installed but it won’t be the latest. So if you want the latest features and bug fixes then you should install the newest version. Generally its a good idea to leave system packages alone because you never know what might depend on them.

Instead you should install Python yourself! The best way to do this is with with pyenv. Pyenv is a utility for managing multiple Python installations and helps to isolate each project from each other. Before pyenv, I used deadsnakes however I find pyenv much easier to setup and use.

Manage multiple Python versions with pyenv

Pyenv can be downloaded from GitHub and placed in your PATH or installed with Homebrew. Homebrew is a system package manager for Linux and macOS — which is my preferred method for installing pyenv because I also install other packages using Homebrew and can keep my packages up to date more easily than manual installations.

Once pyenv is installed the next step is to install one or more Python versions. You can do this using pyenv install $VERSION (where $VERSION is an available Python version).

Now you can run pyenv local $VERSION in any directory to use a specific version of Python within that directory. Of course, any Python packages you install will be associated with that Python installation. If you have projects with different dependencies, this becomes problematic. Thus the next thing to do is create a virtual environment for the project.

Automate venv activation with direnv

Creating a virtual environment is fairly easy using venv. venv is part of the standard library, so no additional installation is required.

However rather than create and active the virtual environment manually, I prefer to use direnv. direnv is a shell extension that automatically loads and unloads environment variables depending on the working directory. Simply, when entering a directory, direnv will automatically load environment variables from a .envrc file and unload them upon exiting. This makes project-specific configuration a breeze.

Additionally direnv has a number of built-in utilities, one of which is layout. The layout command allows you to setup a project using pre-existing configurations. Project layouts are provided for several languages and different configurations, including Python and pyenv.

A typical layout configuration might look like this:

layout pyenv X.Y.Z
export FOO=bar
export BAZ=qux

This will create a virtual environment for the project using the specified Python version.

Once pyenv and direnv are installed, it is easy to configure your project by adding layout pyenv X.Y.Z to the .envrc file in the project directory. As long as the specified Python version is installed (e.g. pyenv install X.Y.Z) then direnv will create (if needed) and automatically activate the project venv when entering the directory. Since direnv automatically loads the environment variables, you can use pip and other tools without having to worry about activating and deactivating the virtual environment.

Alternatives

There are many other tools and approaches to managing Python and Python packages. Data science and machine learning projects often use Conda and pip to manage Python environments and dependencies. However, neither Conda nor pip are designed to manage Python versions. Furthermore, Conda brings a “batteries included” approach by installing a large number of packages by default. This is fine for exploratory data analysis, but not so great for production environments where you want to minimize the number of packages you install.

Many developers use poetry to manage Python environments and dependencies. Poetry is a Python package manager that is designed to be simple and easy to use. It is also designed to be simple and easy to use. However, it does not support managing Python versions. Thus, while I do use poetry to manage dependencies for my projects, it requires an additional installation step and I don’t use it to manage Python versions.

Closing

Managing Python versions and environments is still a challenge for many developers. Developers should be able to install and run a project without affecting other projects or their system.

While there are many tools available to help with this, focusing on isolation and reproducibility is crucial to adopting best practices. My approach first uses pyenv to isolate the project Python environment from the system and then uses direnv to make it easy to reproduce the environment for each project.

I’ve been using this approach for a few years now and it has worked well for coworkers and myself.