ModuleNotFoundError when multiple projects share state database with external macro dependencies

[Tom-Fynes]

Description

When multiple SQLMesh projects share the same state database and one project registers macros with external dependencies, other projects that access the same state database encounter ModuleNotFoundError during the planning phase, even if they don't use those macros.

Setup

We have the following architecture:

• Project A (data-pipeline-alpha): Contains macros that import from an external package shared_utils
• Project B (data-pipeline-beta): A separate SQLMesh project that does NOT have shared_utils as a dependency
• Both projects share the same state database (PostgreSQL)

Steps to Reproduce

1. Create Project A with a macro that imports an external dependency:

# Project A: macros/my_macro.py
from shared_utils import helper_function

@macro()
def my_custom_macro(evaluator):
    return helper_function()

2. Run sqlmesh plan in Project A to register models and macros in the shared state database

3. Switch to Project B (which does not have shared_utils installed)

4. Run sqlmesh plan in Project B

Expected Behavior

Project B should be able to run sqlmesh plan successfully without requiring dependencies from Project A, as long as Project B's models don't reference Project A's macros.

Actual Behavior

Project B fails during planning with:

Traceback (most recent call last):
  File "/path/to/.venv/bin/sqlmesh", line 10, in <module>
    sys.exit(cli())
             ^^^^^
  File "/path/to/.venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  [... stack trace ...]
  File "/path/to/.venv/lib/python3.12/site-packages/sqlmesh/core/model/definition.py", line 1398, in render_query
    query = self._query_renderer.render(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/.venv/lib/python3.12/site-packages/sqlmesh/core/renderer.py", line 570, in render
    expressions = super()._render(
                  ^^^^^^^^^^^^^^^^
  File "/path/to/.venv/lib/python3.12/site-packages/sqlmesh/core/renderer.py", line 161, in _render
    macro_evaluator = MacroEvaluator(
                      ^^^^^^^^^^^^^^^
  File "/path/to/.venv/lib/python3.12/site-packages/sqlmesh/core/macros.py", line 213, in __init__
    prepare_env(self.python_env, self.env)
  File "/path/to/.venv/lib/python3.12/site-packages/sqlmesh/utils/metaprogramming.py", line 548, in prepare_env
    exec(executable.payload, env)
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'shared_utils'

Root Cause Analysis

The error occurs in MacroEvaluator.__init__ at when prepare_env() attempts to execute macro code that contains imports from external dependencies. This happens even when:

• The models in Project B don't use the macros from Project A
• Project B is loading its own models from the shared state

It appears that SQLMesh is attempting to prepare/load ALL macros registered in the state database, not just the ones needed for the current project's models.

Workaround

Current workaround requires all projects sharing the state database to install ALL external dependencies used by ANY project's macros, which defeats the purpose of having separate projects.

Impact

This issue prevents multiple independent SQLMesh projects from sharing a state database if any project uses macros with external dependencies.

This also makes it hard to

• Have a single state management Database
• Share common utilities via external packages
• Maintain separate projects with different dependency requirements