I have a directory structure that look somewhat like this:
.
├── app.py
├── bar
│ ├── app.py
│ └── crawler.py
│ └── database.py
└── foo
├── app.py
└── crawler.py
└── database.py
Here foo
and bar
are self-contained apps that are not (and shouldn't be) aware of the greater structure and they are not packages, but I can add __init__.py
's if I have to. app
and crawler
would import database
, and I can execute each app.py
or crawler.py
from anywhere.
Here comes the problem: I would like to import bar.app
and foo.app
from the outer app.py
. (In case you wonder, foo.app
and bar.app
are both WSGI apps, and the outer app would be a dispatcher, but those details aren't important here.) Apparently import database
wouldn't work since the subdirectories are not on sys.path
, and worse, even if I put the subdirectories on sys.path
, the conflicting database.py
and possibly other modules within subdirectories would mean importing one puts it in sys.modules
, shadowing others with the same name. Also, I can't use relative imports within the subdirectories, or I would run into the __main__
is not a package problem. Again, I don't want to make the subdirectories aware of the greater structure (they should remain independent and be able to freely roam into any directory and still work), so it's hard to make them packages while still maintaining the ability to execute app.py
, crawler.py
, etc. at will.
Right now I have a hack like this:
#!/usr/bin/env python3
import importlib
import pathlib
import sys
HERE = pathlib.Path(__file__).resolve().parent
def import_app(name):
subdir = HERE / name
sys.path.insert(0, subdir.as_posix())
globals()[f"{name}_app"] = importlib.import_module(f"{name}.app").app
sys.path.pop(0)
for mod in subdir.glob("*.py"):
if mod.stem != "app" and mod.stem in sys.modules:
del sys.modules[mod.stem]
import_app("foo") # provides foo_app
import_app("bar") # provides bar_app
Anyone can think of a less hacky solution?
EDIT: To be perfectly clear, the primary goal here is to not complicate the subdirectories as independent projects. The outer app should only serve to transparently dispatch (therefore saving a few worker processes). Therefore, if the solution is to turn the invocation
./crawler.py
into
PYTHONPATH=$PWD/.. python3 -m dirname.crawler
it violates the primary goal.
Comments
Post a Comment