dbt packages: installing, pinning and overriding them

1.What a package adds to your project

A dbt package is a library for analytics code: someone else's dbt project that you install so you don't have to rewrite problems they have already solved. Many analytics problems look the same from one organization to the next. Examples include turning Segment pageviews into sessions, generating SQL to union two relations, building a surrogate key, and writing audit queries. Packages are the dbt form of the [modularity and DRY reuse](lesson:1.3) you practise inside one project, applied across projects.

Installing a package merges it into your project. Every resource in the package becomes part of your project, with four practical effects:

- Models in the package are materialized when you run dbt run. - Your own models can ref the package's models. - Your models can use source to point at sources the package defines. - You can call the package's macros from your own code.

The first effect catches people out. Installing a modelling package such as dbt-labs/snowplow adds its models to your next run, not just a few helper functions.

Two boundaries are worth knowing. dbt packages have nothing to do with Python packages: you define them and install them through a separate mechanism. And packages fit generic problems best. Shared macros, and models for a common dataset such as MailChimp or Snowplow, make good packages. Business-specific logic such as marketing attribution or monthly recurring revenue does not.

Sources12

2.Declaring dependencies in packages.yml or dependencies.yml

Packages are declared in a YAML file that sits at the same level as dbt_project.yml. You list the packages there and then run dbt deps to download them. You can name the file packages.yml or dependencies.yml, and the choice depends on what else the file has to do.

Jinja decides most cases. packages.yml supports Jinja rendering, a capability it keeps for historical reasons. That lets a package specification read env_var, var (only variables passed with --vars), target or dbt_version. It is also the only option when a private Git package uses the older git-token method, where a token from an environment variable is inserted into the URL. Only packages.yml and profiles.yml accept secret environment variables carrying the DBT_ENV_SECRET_ prefix. dependencies.yml deliberately does not render Jinja, which keeps its configuration static and predictable.

packages:
  - package: dbt-labs/dbt_utils
    version: "{{ env_var('DBT_UTILS_VERSION') }}"

dependencies.yml can hold two kinds of dependency. Package dependencies are the library-style source code covered on this page. Project dependencies are the cross-project references used in dbt Mesh, which only dependencies.yml can declare. If you use Hub packages alone, keep packages.yml. If you mix Mesh project references with non-private packages, a single dependencies.yml keeps everything in one place. Native private packages, declared with the private: key, also work in dependencies.yml, because they need no Jinja. If your existing packages.yml contains no Jinja, you can simply rename it.

Sources134

3.Hub, git, local, tarball and private packages

Each entry under packages: starts with a key that says where the code lives, and that key decides which other fields are required. The table pairs each key with what it needs and the situation it suits.

Hub packages must specify a version. Hub releases follow semantic versioning, so the recommended pattern is a range that pins one minor release and accepts its patch updates. Prereleases such as a1, b2 or rc3 are left out of resolution by default. You get one only by naming that prerelease version exactly, or by setting install_prerelease: true together with a range that includes it.

packages:
  - package: dbt-labs/snowplow
    version: [">=0.7.0", "<0.8.0"]

Prefer the Hub when packages share dependencies. Suppose your project uses Snowplow and Stripe, and both of them depend on dbt-utils. Installed from the Hub, dbt resolves that shared dependency once. The other installation methods cannot handle the duplicate dbt-utils. Choose git when you need a branch or a specific commit, and private when the repo is internal. With private, the dbt platform authenticates through its configured GitHub, GitLab or Azure DevOps integration. Locally, on dbt v1.12+ or dbt v2, it uses your SSH key. Either way you don't provision an access token.

packages:
  - git: "https://github.com/dbt-labs/dbt-utils.git" # git URL
    revision: 0.9.2 # tag or branch name

Sources1

4.dbt deps and the package-lock.yml file

dbt deps downloads packages into dbt_packages, and by default Git ignores that folder, so the package source code isn't duplicated in your repository. To install somewhere else, set packages-install-path in dbt_project.yml, for example packages-install-path: packages.

package-lock.yml is what makes installs repeatable. On its first run, dbt deps writes a lock file in the project root. It records the exact resolved version of every package, including commit SHAs. Later runs install from the lock file. As long as your packages file is unchanged, every developer and every environment gets the same versions, even after newer releases appear. That stability is the reason to commit package-lock.yml to version control. If you git-ignore it instead, builds become less predictable, and you should then pin third-party package versions in the packages file itself.

The lock file contains a sha1_hash of your packages config, which is how dbt notices edits. If you add a package or widen a range, the next dbt deps resolves the dependencies again and rewrites the lock. Three flags change this default behaviour:

- --upgrade refreshes every package even when the packages file is unchanged. This is useful for pulling the latest commits from the main branch of an internal Git package, though forced upgrades can cause build inconsistencies. - --lock updates the lock file to match config changes without installing anything. - --add-package writes the entry for you, for example dbt deps --add-package dbt-labs/dbt_utils@1.0.0. Hub is the default type; add --source git or --source local for the other two.

dbt deps  ? 

Sources567

5.Overriding package macros with dispatch

Packages that run on many warehouses rely on dispatch. A generic macro hands the work to an implementation chosen for the current adapter. dbt searches for macros named with the adapter prefix, such as snowflake__concat or redshift__concat, and falls back to default__concat when no adapter-specific version exists. Dispatched macros inside a package pass a macro_namespace, for example adapter.dispatch('concat', 'dbt_utils'). That namespace tells dbt which package to search.

That namespace lets you override a package without forking it. Writing your own concat macro changes only your own calls to it. Macros inside dbt_utils, such as surrogate_key, still call dbt_utils.concat. To route those calls through your version as well, add a dispatch config with a search_order for the dbt_utils namespace. The config only works in your root dbt_project.yml; dbt ignores dispatch configs in the project files of installed packages.

dispatch:
  - macro_namespace: dbt_utils
    search_order: ['my_project', 'dbt_utils']

dbt only ever considers implementations that are compatible with the current adapter, and within those it finishes searching one package before moving to the next. On Postgres the lookup is my_project.postgres__concat, then my_project.default__concat, then dbt_utils.postgres__concat, and finally dbt_utils.default__concat, which is found and used. The same pattern lets an internal helper package override dbt's own global macros, such as generate_schema_name: each root project lists that package ahead of dbt in the search_order for the dbt namespace. Some things cannot be overridden this way. ref, source and config are context properties, not dispatched global macros, so dispatch never reaches them.

Sources8

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Installing dbt-utils from git handles shared dependencies just as well as installing it from the Hub.Why is that wrong?

   Only Hub installation lets dbt resolve a dependency shared by several packages. Git and the other methods cannot handle the duplicate dbt-utils.

   Covered in Hub, git, local, tarball and private packages

2. 2.Every dbt deps run fetches the newest version allowed by your version range.Why is that wrong?

   When the packages file is unchanged, dbt deps installs the versions recorded in package-lock.yml. Newer releases come in only after a config change or with --upgrade.

   Covered in dbt deps and the package-lock.yml file

3. 3.A dispatch config can replace how ref or source behave for an installed package.Why is that wrong?

   ref, source and config are context properties, not dispatched global macros, so dispatch cannot override them.

   Covered in Overriding package macros with dispatch

4. 4.A private Git package that uses an environment-variable token can be declared in dependencies.yml.Why is that wrong?

   The git-token method relies on Jinja, which dependencies.yml does not render, so those packages must go in packages.yml.

   Covered in Declaring dependencies in packages.yml or dependencies.yml

Practise it for real

Add dbt-utils to a project, and leave it pinned and reproducible for the rest of the team.

1. 1.Run dbt deps --add-package dbt-labs/dbt_utils@1.0.0 from the project root.

   Why: The command writes the package entry for you, so you don't have to remember the exact syntax. Hub is the default type.

   You should see: The package appears in your packages file and is installed into the dbt_packages directory.

2. 2.Open package-lock.yml in the project root.

   Why: This file is what keeps every environment on the same versions.

   You should see: It records the exact resolved version of dbt_utils and a sha1_hash of your packages config.

3. 3.Commit package-lock.yml and the packages file. Leave dbt_packages out of the commit.

   Why: Teammates and jobs will install from the lock file, and Git ignores dbt_packages by default.

   You should see: A teammate who runs dbt deps gets exactly the same dbt_utils version.

4. 4.Change the version in the packages file and run dbt deps --lock.

   Why: This shows how dbt detects a config change without installing anything.

   You should see: The lock file is updated to the new resolution, and no packages are installed.