Installing Python, virtualenv, NumPy, SciPy, matplotlib and IPython on Lion

I decided to clear the cobwebs out of my  MacBook Pro (update 8/20/11: new MacBook Air in hand!) and start fresh with Lion, Python and all the other tools I regularly use (virtualenv, NumPy, SciPy, matplotlib and IPython). It was a lengthy process and I was unable to find a comprehensive walkthrough online, so I had to figure it out through trial and error (mostly error). I was surprised at the lack of available information; with such a major release I expected more complete documentation. So, once again quoting Tom Lehrer, I have a modest example here.

The important things we'll accomplish in this guide are (click any section to jump directly there):

1. A clean install of Mac OS 10.7 Lion
2. Installing Homebrew
3. Installing Python (and pip)
4. Installing virtualenv
5. Installing NumPy, SciPy and matplotlib
6. Installing IPython and the qtconsole

As you may recall, this has been a somewhat more-than-trivial process for Lion's early adopters, as enough was changed behind the scenes to mess up some of the current versions of these tools. I'll detail how to get around those issues by getting cutting-edge distributions. Note that today's cutting-edge works on my machine; tomorrow's may introduce new bugs. When stable Lion-compatible versions of these programs are available, I strongly encourage you to use those instead.

If you would rather skip my commentary, you should be able to simply execute every line of code in sequence.

As always, remember that some of these lines will take you off the beaten path and could cause serious irrevocable damage to your computer! In particular, typing sudo gives you root access and therefore the ability to erase everything with a single keystroke. Please do not execute any code without understanding exactly what it does. In the worst case, I suppose you could always jump back to square one by wiping your hard drive and running a Lion clean install, but use caution nonetheless. YMMV.

Lion Clean Install

To begin, there has been some concern that users can't perform a clean install of Mac OS X 10.7 (Lion) without quite a bit of hacking around with physical media. I'm happy to report that isn't true -- as long as you performed a standard Lion install first. The OSX installer (App Store link) automatically creates a "Recovery HD" partition on your hard drive, just in case something goes wrong. The partition contains just two things: Disk Utility  and the ability to run the Lion installer. To access this recovery mode, hold Command-R while your computer is starting up. Use Disk Utility to erase your hard drive, then run the installer to re-download and load Lion. When you're done, you'll have a clean install of Lion. Download your favorite browser and you're good to go. I have to say, discovering this feature was a pleasant surprise -- hats off to Apple for thinking ahead on this one.

Next, install Xcode 4 -- its bundled compilers have to be available for the next step to work. Here's a link to it in the App Store; it's a free (but large) download. Note that like Lion, the download does not actually install the application. Instead, it puts an "Install Xcode" icon in your applications folder, which you must run to complete the installation. I don't understand this behavior of installing the installer, but it is what it is.

Homebrew

Next up, one of my new favorite tools -- Homebrew. The homepage says it pretty well: "Homebrew is the easiest and most flexible way to install the UNIX tools Apple didn't include with OS X." If you've messed around with MacPorts (or tried to do this sort of stuff  -- gasp -- by yourself), you'll be shocked by the drop-dead simplicity of this tool. Even installing it is a no-brainer; just fire up Terminal and execute the following line:

/usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"

That's it -- now you have Homebrew. Try running brew update and brew doctor at the command line to make sure it's working properly.

You'll need to add the Homebrew directory to your system path, in order to make sure that Homebrew-installed software is given higher priority than any other version. To do so, open your .bash_profile in your user directory (that's the ~ directory or, more verbosely, /Users/[your_user_name]) and add the following line (if you don't have a .bash_profile, create one with your favorite text editor -- you may need to do this in order to see it, as it will be a hidden file):

export PATH=/usr/local/bin:$PATH

If you haven't already, restart Terminal so it picks up the new path. Homebrew is generally very good about alerting you to any action you need to take after it finishes running, including path modifications. Make sure to pay attention to its output.

Python

Now on to the main event: Python. Let's start by installing some prerequisites (the last one is a prerequisite for IPython, but we might as well get it with the others):

brew install readline sqlite gdbm pkg-config

And now Python itself. As of this writing, 2.7.2 is the most recent version. There are some version-specific references in the following instructions; change them as necessary. The --framework option tells it to build as a Framework, which has some downstream niceties, and --universal builds a universal (32/64 bit) version:

brew install python --framework --universal

This will install a version of disutils, but you must update your path in order to run it properly. Add the following to your .bash_profile (you may combine it with the previous .bash_profile edit, if you choose):

export PATH=/usr/local/share/python:$PATH

Once again, make sure you reload Terminal to update your path variable. Finally, you need to change Lion's symlink to point at your new Python install. Run the following three lines in sequence:

cd /System/Library/Frameworks/Python.framework/Versions
sudo rm Current
ln -s /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/Current

You will be prompted for your password after executing the sudo command.

To verify that your installation went as planned, type which python at the command line. You should see /usr/local/bin/python in response. Furthermore, if you type python, you should see Python 2.7.2 launch (type quit() to return to Terminal).

You already have easy_install available through Homebrew's version of disutils, but pip, in my opinion, is a much better package manager -- not least because it allows simple uninstalls. To get pip, simply execute:

easy_install pip

Note that depending on your permissions settings, you may have to prepend sudo to the front of many of these lines. Always remember that sudo gives you root access!! Therefore, in the interest of caution, I'm not going to include the sudo command here, but if your installs are failing because of permissions, you may have to add it.

virtualenv

Virtualenv is a tool that allows you to create isolated Python environments, each with its own set of packages and dependencies. This is useful for testing or managing package requirements (for example, if you build an application that is dependent on a certain version of a third-party package but another application requires a more recent version, you may break the first application by upgrading). This is not a required step, and all the commands below should work whether or not you are using virtualenv, so I'm listing it here for convenience only. The only difference will be that directories (such as that returned by which pip) will point to the virtualenv rather than /usr/local.

First, install virtualenv and virtualenvwrapper (a tool which makes working with virtualenv somewhat easier):

pip install virtualenv
pip install virtualenvwrapper

Next, source the virtualenvwrapper script:

source /usr/local/share/python/virtualenvwrapper.sh

This will create a hidden virtualenv directory at ~/.virtualenv. Now you can make your first virtual environment:

mkvirtualenv test1

Your new virtualenv test1 comes with a complete install of Python 2.7.2 and its own version of Pip. It is activated by default, so running any pip command will only impact this environment. Note that if you deactivate the virtualenv, you will lose access to any packages installed in it. You can switch between virtualenvs with the workon command.

In order to delete your test virtualenv, run rmvirtualenv test1. Also, to specify a different version of Python, just use mkvirtualenv -p python3.2 test1 (replacing python3.2 with the shell name of your preferred version).

NumPy

If pip installed properly (executing which pip should return /usr/local/share/python/pip), then you can install NumPy simply with:

pip install numpy

Check your work by executing:

python

followed by:

import numpy
print numpy.__version__
print numpy.__file__
quit()

The version should be 1.6.1 (as of this writing), and the file should be somewhere in /usr/local/Cellar/....

SciPy

SciPy requires a Fortran compiler, which Lion lacks. To acquire one, use Homebrew:

brew install gfortran

As of this writing, the stable version of SciPy (0.9) will not compile properly under Lion. If you try installing SciPy with pip, it will fail. Fortunately, a fix has already been committed to the dev version (0.10) and we can instruct pip to use that version:

pip install -e git+https://github.com/scipy/scipy#egg=scipy-dev

As with NumPy, make sure you have everything working by running:

python

and then:

import scipy
print scipy.__version__
print scipy.__file__
quit()

Technically speaking, it still isn't compiling properly -- but this should run well enough for most work. If you're a stickler for these things, then you can force SciPy to compile using gcc 4.2, rather than Lion's default llvm-gcc 4.2, by reassigning the CC shell variable to the appropriate compiler and building the package. Note that if you want to do this with Homebrew, you'll need to adjust the symlinks to cc, gcc, g++ and c++ in /usr/bin to point at gcc-4.2 (and its variants as appropriate) rather than their current targets, because the Homebrew script calls the symlink rather than the shell variable. I'm going to assume that if you choose to go this route, you also have the know-how to do so, but feel free to email me otherwise. Note that this method isn't necessarily "correct" and I'm not a SciPy developer. Most likely, this will all be handled automatically by the final 0.10 release.

matplotlib

Like SciPy, matplotlib doesn't play nicely with Lion (due to an incompatibility with libpng 1.5). By this point, you know the drill -- just point pip at the patched development version, and check its work when it finishes:

pip install -e git+https://github.com/matplotlib/matplotlib#egg=matplotlib-dev

IPython

Installing IPython itself is a fairly straightforward pip command:

pip install ipython

Getting the new qtconsole to run takes a little more work -- but it's well worth it. The qtconsole is like running IPython on steroids -- my favorite feature is that matplotlib output can be displayed inline.

To begin, you'll need Nokia's qt library, which is available here. Make sure you download the library, not the SDK.

Once that's done, begin installing the prerequisites:

brew install pyqt

Note that after installing pyqt, brew will prompt you to add the following to your pythonpath in .bash_profile:

export PYTHONPATH=/usr/local/lib/python:$PYTHONPATH

Continue installing:

brew install zmq
pip install pyzmq
pip install pygments

And that's it -- you should now be able to launch the qtconsole by executing ipython qtconsole. If you want to see the matplotlib output inline (and why wouldn't you?), then execute:

ipython qtconsole --pylab=inline

Fin.

Phew! I hope this works for everyone out there. Until the next software upgrade, anyway.

Updated 9/1/11 to include virtualenv.

Tagged as: code, ipython, matplotlib, numpy, python, scipy