Adding dbt projects to a mesh

1.When a second project is worth it, and where to cut

Add a second dbt project when a single project is causing problems for the organisation, not as a starting setup. The usual signs are these. The model count is slowing development. Teams' workflows have drifted apart and need to be separated. Data products are becoming less reliable because teams struggle to communicate. Security or governance requirements would benefit from isolation. A team just starting with dbt does not need any of this yet. The features behind a multi-project setup work on their own and can be adopted as the project grows.

Cross-project references are the feature that turns several projects into one mesh. They are available only on the dbt Enterprise and Enterprise+ plans. On any lower plan you can still have separate projects, but a ref cannot reach from one to another.

The hardest decision is where to draw the line in the DAG. A vertical split separates layers of transformation. Examples are a shared staging project that others build on but cannot edit, an early project that masks PII before anyone downstream sees it, and an isolated set of expensive models that can't be selected by accident. A horizontal split separates by domain or source. Examples are the marketing team's data flow, clickstream events kept apart from transactional data, and two teams that work at different speeds. You can combine the two kinds, but pick one to do first. In a hub-and-spoke organisation, break out the central platform project before splitting the rest into domains.

Good boundaries usually exist already. Look at which models your jobs build together, how the lineage graph clusters, and which resource groups people have defined in selectors.yml. A team that sits entirely downstream of the current project is often the easiest first consumer.

Since Mesh launched, the most common pattern has been one project per team, each in its own repository. Some contributions across projects are healthy. If teammates need a coordinated change across several projects every week, or for a fifth or more of their work, the boundaries are in the wrong place.

Sources123

2.Creating the new project in dbt

Every project in a mesh is a complete dbt project with its own data platform connection, git repository and environments. To create one, go to Account settings, then Projects, and click New project. Enter a name, pick the data platform and fill in the connection settings, then click Test Connection before continuing. As a running example, take a fictional company that creates two projects this way: an upstream "Jaffle | Data Analytics" and a downstream "Jaffle | Finance", with the default databases jaffle_da and jaffle_finance. Connection testing has its own lesson: [testing a project connection](lesson:1.3).

A project can join a mesh only once it has a deployment environment set as Production or Staging. Each project can have one deployment environment designated as Production. To create it, go to Deploy → Environments → Create New Environment, choose Deployment as the type, then select Production under Set deployment type. Which environment type cross-project references resolve against is covered in [environment types and cross-project references](lesson:7.2).

Create a new project for a new codebase, never for another stage of the same codebase. Project names must be unique within a dbt account, and cross-project references find projects by name. Differences between dev and prod belong in environments. Each environment can use a different connection, connections can be reused across projects and environments, and one project can use several connections of the same warehouse type. That removes the usual reason teams create a duplicate project.

Two things must be true before a downstream team can build on a new foundational project. At least one model must be defined as public, and a deployment job must have run successfully. That run produces the manifest.json file that downstream projects resolve their references against.

Sources45

3.Moving models into the new project

Before splitting anything, set up groups and access inside the monolith. Use groups to assign owners and set each model's access as tightly as current behaviour allows. Then run each group with selection syntax alongside your production jobs until you trust the boundaries. These settings are covered in [model governance](lesson:7.3). Once the boundaries are confirmed, splitting off one group follows a fixed checklist:

1. Move the group's models into a subfolder, along with their YAML entries and any parent or child models that belong with them. 2. Create a new dbt_project.yml in that subfolder. 3. Copy any macros the moved models use. 4. Create a new packages.yml there for the packages those models need. 5. Change every ref that crosses projects to two arguments, project name then model name. Set the upstream models to access: public, ideally with a model contract. A contract sets explicit expectations on the shape of the data, so changes upstream do not break the downstream consumers who rely on it. 6. Add a dependencies.yml to the downstream project that names the upstream project.

# in dependencies.yml
projects:
  - name: jaffle_shop

Discipline matters more than speed here. Split one group at a time and aim for exact 1-to-1 parity with the old project. Note improvements for later instead of refactoring during the migration. Start inside the same repository, so git history gives you an easy way to undo the split. Circular references are still not allowed, and across projects the test is made per model: two projects may depend on each other in both directions, but a cycle exists when a model ends up depending on itself through models in either project. dbt checks for such cycles and raises an error.

Some organisations already run several projects. They either install the parent project as a dbt package or read its outputs with source(). Both approaches have costs. A package puts every upstream resource into the child's manifest, which slows dbt down. Sources break the lineage, because nothing connects the two projects. Moving either setup to the mesh is simpler than splitting a monolith. In the parent, mark the referenced models public and add contracts. In the child, add the parent to dependencies.yml. Then, if you used a package, remove its packages.yml entry and add the project name to each ref. If you used sources, replace each source() with a cross-project ref and delete the source definitions you no longer need.

Sources315

4.Declaring the dependency and writing the ref

Packages and projects are two different kinds of dependency, and only project dependencies are part of the mesh. A package brings in its full source code, like a library. A project dependency brings in no code. The downstream project treats each upstream public model as an API that returns a dataset, and the model's maintainer is responsible for its quality.

dependencies.yml sits at the root of the project and can hold both kinds of dependency: projects go under projects: and Hub packages under packages:. Keep using packages.yml when your package specifications use Jinja, for example a Git token read from an environment variable. dependencies.yml does not render Jinja and does not support private packages. The project name must match the upstream dbt_project.yml exactly, including case: jaffle_marketing, not JAFFLE_MARKETING.

packages:
  - package: dbt-labs/dbt_utils
    version: 1.1.1

 ? :
  - name: jaffle_finance  # case sensitive and matches the 'name' in the 'dbt_project.yml'

Every reference to a model from a project dependency uses the two-argument form, with the project name first. A one-argument ref looks only inside the current project. The upstream model must also be access: public. The default access is protected, which allows references only from within the model's own project.

select * from {{ ref('jaffle_finance', 'monthly_revenue') }}

Once you commit the change, dbt resolves the dependency, makes sure upstream projects are built first, and shows lineage across projects in the DAG. If the Finance team later renames the model, changes its schema or bumps its version, the ref still resolves. It points at wherever Finance builds the model in production.

Sources54

5.Projects that depend on each other

Two projects can depend on each other in both directions, as long as no individual model ends up in a cycle. Finance can use Marketing's public models while Marketing uses Finance's. dbt checks for cycles across projects and raises an error if it finds one.

A pair like this has to be set up step by step. Neither project can resolve a reference until the other has published public models from a deployment run. If you add both dependencies at once, whichever project runs first has nothing to resolve against.

Sources5

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.You should give one team separate dbt projects for dev and prod, such as "Marketing - Dev" and "Marketing - Prod".Why is that wrong?

   Project names must be unique in the account. Separate dev and prod with environments, each of which can use its own connection.

   Covered in Creating the new project in dbt

2. 2.Installing the upstream project through packages.yml sets up a mesh dependency.Why is that wrong?

   A package pulls in the full source code and is not part of the Mesh workflow. Cross-project references need a project entry in dependencies.yml.

   Covered in Declaring the dependency and writing the ref

3. 3.Once the dependency is declared, a one-argument ref('monthly_revenue') finds the upstream model.Why is that wrong?

   Models from a project dependency must be referenced with the two-argument form that names the project.

   Covered in Declaring the dependency and writing the ref

4. 4.Project names in dependencies.yml are matched without regard to case.Why is that wrong?

   The name must match the upstream dbt_project.yml exactly, including case.

   Covered in Declaring the dependency and writing the ref

Practise it for real

Make a downstream project reference a public model from an upstream project through a project dependency.

1. 1.In the upstream project, set access: public on the model you want to share, merge the change, and run a deployment job in the Production environment.

   Why: Downstream references resolve only to public models, using metadata from a successful deployment run.

   You should see: The job succeeds and the model shows as Public in the lineage view.

2. 2.In the downstream project, create dependencies.yml at the root and list the upstream project's name under projects:.

   Why: This declares a project dependency instead of installing the upstream code as a package.

   You should see: The name matches the upstream dbt_project.yml exactly, including case.

3. 3.In a downstream model, select from the upstream model with a two-argument ref: project name first, then model name.

   Why: Project dependencies require the project name in the ref.

   You should see: The model compiles against the upstream model's production location.

4. 4.Commit the changes and check the downstream lineage.

   Why: Once the dependency is configured, dbt resolves it and shows lineage across projects.

   You should see: The upstream public model appears as a parent in the DAG.