Managing Multiple Python Installations
With the recent release of Python 3.7.0, I’ve found myself in a position I was warned of early on when I started using Python, but thought I had solved with virtual environments. I was sorely mistaken when my PyTorch GPU build broke after upgrading Python 3.6.5 with Homebrew. Virtual environments are Python’s answer to the frustrating problem of package management. Different projects and utilities often rely on particular versions of packages or modules that potent. Virtual environments allow different versions of the same package to live happily together on the same system, partitioned in their own little universe that includes an independent Python installation as well. It also has the advantage of encapsulating more obscure dependencies with their files for organization (Just run pip freeze and try to guess what goes with which project). Great! We’ve got packages sorted out, what about Python installations themselves? This is where things get a little murky. First of all, it goes without saying that you should never ever mess with the system’s installation of Python (Located in /usr/bin/python on macOS). There are plenty of ways to install an additional binary. Installer packages as well as source release archives are available directly. As you may already have guessed, clever reader, there are a few reasons this is less than ideal, and you’re right! Building from source is a hassle and unless your a masochist like me, or have system optimizations (Like GPU support *cough* PyTorch *cough*), it’s generally more trouble than is worth. The installer packages, on the other hand, are stupid easy to install, but add unnecessary (And unwanted) cruft to your Applications folder. Homebrew, however, will install python (2 and/or 3) alongside your system’s installation, and update your PATH variable (More on that below) so they all play along nicely. What’s not to love?
Installing multiple versions of Python with Homebrew
Homebrew is great for installing the latest version of Python, but anything else requires diving into git to checkout the correct commit. For example, this is how to install Python 3.6.5_1:
And to add insult to injury, the commit history is too long to generate online, so one would need to clone the entire repository to access the history. Luckily, there is a better way. Enter Pyenv. A fork of the Ruby evnvironment manager rbenv, pyenv works the same way, by injecting shim executables into your PATH. What?
Follow the Yellow/Blue Brick Road
The PATH environment variable has been shrouded in mystery from me for far too long than I care to admit, but it’s way simpler than it’s made out to be. PATH is just a list of paths to directories (Hence the name) for the operating system to search through when looking for an executable, from first to last, separated by colons. That first caveat is an important one. For example, calling python in a shell returns the REPL. If I have two Python installations in my PATH, the one that’s found first is the one used. If one installation is my system installation and is in /usr/bin, and the other I’ve installed to /usr/local/bin, I’ll want to export my PATH with /usr/local/bin ahead of /usr/bin if it isn’t already (Though it usually is).
Pyenv’s shims are injected at the head of the PATH variable so the will intercept any calls to python, and direct them to the appropriate version’s binary through a symlink, as described by the current directory’s .python-version file, or the default if no such file exists. You can set the desired version with:
To install pyenv, simply run:
Once installed, you can get a list of available Python versions with:
And install with:
To see a list of all installed Python versions:
Venv and virtualenv and pipenv, oh my!
In the beginning, there was virtualenv, a tool for creating isolated Python environments. It’s an indispensible tool if more than one project is being developed on the same system, as it allows each project to install its own copies of its dependencies. This allows for multiple versions of the same dependency to exist on a system without fear of name collision and helps keep the global package namespace clean. Virtualenv is so useful that in Python 3 it was folded into the standard library as venv. If you don’t need or care about Python 2 support, then you can use venv out of the box and have all the features of virtualenv. Pipenv is the latest iteration of virtual environment tools, and the recommended dependency manager for applications. While Virtualenv and venv are great, they fall short in certain use cases, such as collaborative or version-controlled projects. Pipenv bakes in pip and Pipfile, simplifying the installation process (Pip must be manually installed in new environments created by virtualenv and venv), and installing the virtualenv directory in the user’s home directory (~/.local/share/virtualenvs/), as opposed to the current working directory. Additionally, the required dependencies are listed in a Pipfile, a more robust alternative to requirements.txt, and is analogous to packages.json in npm.
Developing in multiple versions of any language is no trivial task, but for Python, with the right tools, it’s become easier than ever.