Using python -m to invoke modules as scripts

Note

Hey there! This gist has been rewritten in my GitHub blog. You should read that instead!

Prologue

You might have seen the -m flag used in various python commands online, or was told by someone else to use python -m to "run a module as a script", but didn't really understand what that meant. This gist covers how that flag is used, particularly in the context of package development.

Introduction to modules and packages

Say you wanted to write a command-line utility for downloading a GitHub repository release, and you started out with a single script, downloader.py. Eventually your code started growing too big to be well-organized in a single file, so you decide to split its functionality into separate modules:

api.py    # interacts with GitHub's API using the requests library
cache.py  # stores release information to avoid unnecessary requests
cli.py    # provides an argparse interface for the user
main.py   # the entry script that your users should run
    from api import download
    from cli import parser

    args = parser.parse_args()
    download(args.repository, args.filename)

This is reasonably organized, but if you wanted to share this to other people or use it in another project, they would need to download all four scripts inside whatever working directory they might be in, as well as any dependencies required by your script:

my_project/
    api.py, cache.py, cli.py, main.py
    # /py_project $ pip install requests
    # /my_project $ python main.py ...

This is pretty inconvenient for developer experience. For writers that want their tools to be as easy as possible to install, they will upload their code on PyPI and configure it so that you can install their package with a single command:

pip install my-fancy-github-downloader
python -m my_downloader

If you want to do the same thing, the first step would be organizing your code into a package, where you've collected your scripts into a single directory:

my_project/
    my_downloader/
        __init__.py
        api.py, cache.py, cli.py
        main.py
            # In packages you can use relative imports:
            from .api import download
            # Though absolute imports are also valid:
            from my_downloader.cli import parser

This way, when someone downloads and installs your tool, it's already contained in one directory and they don't have to worry about accidentally losing or overwriting one of your scripts when they install something else.

How does -m play into this?

With your code being organized as a package, how do you run main.py? You could try python my_downloader/main.py, but this uses a relative file path which relies on your package being in the terminal's current directory. This also means other users would have to remember where they downloaded your package and write out the appropriate file path, which is yet another inconvenience. But most importantly, you lose features that packages normally have like relative imports:

/my_project $ python my_downloader/main.py
Traceback (most recent call last):
  File "/my_project/my_downloader/main.py", line 2, in <module>
    from .api import download
ImportError: attempted relative import with no known parent package

To solve the last issue, you need to tell Python that main.py is part of the my_downloader package and not a standalone script, using -m:

/my_project $ python -m my_downloader.main
# Equivalent to a regular python script containing:
import my_downloader.main
# ...except that the imported module's __name__ constant is set to "__main__"

Now, before Python starts executing main.py, it can load the my_downloader package and allow importing its submodules through the my_downloader name or . relative imports.

You can additionally simplify python -m my_downloader.main by renaming the main.py script to __main__.py. With this change, Python will implicitly run that script when you use -m on the package directly:

my_project/
    my_downloader/
        __init__.py
        __main__.py  # same contents as main.py
        ...

# Now both of the following are valid:
/my_project $ python -m my_downloader
/my_project $ python -m my_downloader.__main__

Now your package can now be "invoked as a script" without relying on a file path. The package still needs to be in the current directory since it isn't "installed", but now you can proceed with writing project metadata to make your package installable + uploadable to PyPI, and defining a setuptools entrypoint to replace python -m my_downloader¹.

Sidenote: why is -m recommended on Windows?

Searching online, you'll find a dozen ways to invoke Python on the command-line (python, python3, python3.11, py, etc.). Beginners to this (especially to the command-line) may not understand how these commands are provided by the PATH environment variable. If they take the shortest path through the official installer, their system's PATH will not be updated to include python or any package entrypoints like pip. However, the installer does include the Python Launcher for Windows by default, providing the py command to invoke python. With py alone, you can still access pip or other installed modules by running their modules directly, e.g. py -m pip install .... If you already understand how your Python installation is set up you won't need to use py -m, but for novices this is typically more fool-proof than asking them to re-install with the "Add Python to PATH" option and potentially confusing them if they have multiple Python versions.

Footnotes

1. See setuptools + pyproject.toml ↩