Paths in Python

How do you get your Python interpreter to find modules that are not located in your current working directory? The answer is … you tell it where to look.

When you type something like

from my_module import my_function

Python searches a collection of directories (i.e., folders) on your computer. If the directory containing <my_module.py> is not in this collection, you will receive an ImportError. This can be frustrating if you know the file exists, and even more so if you know where it exists.

In this post, we will take a brief look at how to add paths to the collection of directories searched by Python.

Paths

A path is essentially a set of directions to a file:

/Users/username/modules/my_module.py        [Mac OS X, Linux]
C:\modules\my_module.py                     [Windows]

It tells your operating system how to navigate from a fixed starting point — the “root directory” / in Unix-based systems, or C:\ in Windows — through a collection of folders to the desired file.

In this case, there is a folder called <modules/> that contains the desired file, <my_module.py>. Let’s suppose that you are writing a script called <my_script.py> in your working directory:

/Users/username/scratch/my_script.py        [Mac OS X, Linux]
C:\scratch\my_script.py                     [Windows]

You want Python to import <my_module.py> when you run the script, but it cannot find the file.

Below are 4 options for adding the path for your module to Python’s collection of paths. None are extremely complicated, but the last two require working with the command line and editing startup files on your operating system. If you are unfamiliar or uncomfortable with this, you might prefer the first two methods.

You will need to replace directory names like /Users/username/ or C:\scratch\ with the correct directories on your own system for these examples to work.

Using The sys Module

There is a module called sys that lets you modify the collection of paths from within Python. According to its documentation,

This module provides access to some objects used or maintained by the
interpreter and to functions that interact strongly with the interpreter.

One of these objects is a list called sys.path. Again referring to the documentation,

path -- module search path

All we need to do is add the path to <my_module.py> to sys.path. On Unix-like systems, the commands are

import sys
sys.path.append('/Users/username/modules')

On Windows, it is easier to use a raw string to specify the path:

import sys
sys.path.append(r'c:\modules')

This will only affect the path in the current Python session. If you quit and restart, you will need to add the path again.

Using Spyder

If you are using the Spyder IDE, you can permanently add a directory to its collection of paths. Chose the “PYTHONPATH manager” from the menu (python > PYTHONPATH manager) as shown below:

Managing paths within Spyder.

This will open a window that shows a list of directories added to Spyder’s collection of paths. If you have not modified this list already, it will probably contain your working directory and nothing else. At the bottom of this window, there is an icon for “Add Path”. Click on this. Now you can find the directory you want to add, as if you were opening a file.

In this case, we navigate to

/Users/username/modules

and select “Choose”.

Once you have selected a directory, it will become a permanent addition to Python’s paths within Spyder. You will need to quit Spyder and restart or “Restart Kernel” for the changes to take effect.

This is a convenient way to access modules that are not part of the Anaconda distribution from within Spyder.

Using the Command Line

If you are working from a command line, you can add your module directory to Python’s collection of paths by setting the PYTHONPATH variable before you start a Python session or run a script:

$ export PYTHONPATH=${PYTHONPATH}:/Users/username/modules
$ python my_script.py

No matter what directory you are working in, Python now knows where to find <my_module.py>.

If you want to permanently add your module directory to PYTHONPATH, you can add the export command above to your <.profile> or <.bashrc> file.

Startup Files

Another option is to create a Python startup file — a Python script that will be run at the beginning of every interactive Python session (but not when you run scripts from the command line). First, create a file such as <python_startup.py> in your home directory. Then, enter Python commands exactly as you would type them at the command prompt:

# python_startup.py
# Startup script for interactive Python sessions.

import sys
sys.path.append('/Users/username/modules')

Next, edit your <.profile> or <.bashrc> file and add the line

export PYTHONSTARTUP=/Users/username/python_startup.py

And that’s it. Now, every time you start a new Python session, you can access your own modules.

IPython does not use the PYTHONSTARTUP variable. If you want to add a startup script for IPython, place a file like <python_startup.py> in IPython’s directory of startup scripts:

/Users/username/.ipython/default_profile/startup/

Now your modules will be available in every IPython session, including Spyder.

Summary

There are several ways to show Python where to find modules. For a single session, appending a path to the sys.path variable is an easy solution. For a permanent change, you can use Spyder’s PYTHONPATH manager or edit startup files from the command line. These methods will allow you to access modules without copying or moving files into your working directory, and you can easily access modules you write yourself or modules you download from the Web.