Donate. I desperately need donations to survive due to my health

Get paid by answering surveys Click here

Click here to donate

Remote/Work from Home jobs

Import from subdirectories without awareness of subdirectories

I have a directory structure that look somewhat like this:

.
├── app.py
├── bar
│   ├── app.py
│   └── crawler.py
│   └── database.py
└── foo
    ├── app.py
    └── crawler.py
    └── database.py

Here foo and bar are self-contained apps that are not (and shouldn't be) aware of the greater structure and they are not packages, but I can add __init__.py's if I have to. app and crawler would import database, and I can execute each app.py or crawler.py from anywhere.

Here comes the problem: I would like to import bar.app and foo.app from the outer app.py. (In case you wonder, foo.app and bar.app are both WSGI apps, and the outer app would be a dispatcher, but those details aren't important here.) Apparently import database wouldn't work since the subdirectories are not on sys.path, and worse, even if I put the subdirectories on sys.path, the conflicting database.py and possibly other modules within subdirectories would mean importing one puts it in sys.modules, shadowing others with the same name. Also, I can't use relative imports within the subdirectories, or I would run into the __main__ is not a package problem. Again, I don't want to make the subdirectories aware of the greater structure (they should remain independent and be able to freely roam into any directory and still work), so it's hard to make them packages while still maintaining the ability to execute app.py, crawler.py, etc. at will.

Right now I have a hack like this:

#!/usr/bin/env python3

import importlib
import pathlib
import sys


HERE = pathlib.Path(__file__).resolve().parent


def import_app(name):
    subdir = HERE / name
    sys.path.insert(0, subdir.as_posix())
    globals()[f"{name}_app"] = importlib.import_module(f"{name}.app").app
    sys.path.pop(0)
    for mod in subdir.glob("*.py"):
        if mod.stem != "app" and mod.stem in sys.modules:
            del sys.modules[mod.stem]


import_app("foo")  # provides foo_app
import_app("bar")  # provides bar_app

Anyone can think of a less hacky solution?

EDIT: To be perfectly clear, the primary goal here is to not complicate the subdirectories as independent projects. The outer app should only serve to transparently dispatch (therefore saving a few worker processes). Therefore, if the solution is to turn the invocation

./crawler.py

into

PYTHONPATH=$PWD/.. python3 -m dirname.crawler

it violates the primary goal.

Comments