Raw object dependencies with source() and ref()

1.Why raw objects need declaring

Every dbt project starts from data that dbt did not build. Extract and Load tools write it into the warehouse, and because third parties load it, its shape changes without warning: tables and columns get added, removed or renamed. If ten models each hard-code raw.jaffle_shop.orders, a rename upstream means editing ten files. If they all go through one declared source, it means editing one YAML entry.

The recommendation is blunt. Define raw data as sources and select from the source, never from a direct relation reference such as my_schema.my_table. A well-kept dbt project has no direct relation references in any model. A hard-coded name still runs, which is why the habit lasts, but dbt cannot see it. It adds no edge to the DAG, the model has no recorded raw dependency, and nothing ties it to the tests or freshness checks defined on that data.

Sources also set where the first transformation layer begins. Each staging model should select from only one source. It renames fields to the project's conventions (for example, every timestamp named <event>_at) and recasts types. Everything downstream builds on those staging models, so each raw object is touched in exactly one place. How that layering carries on through intermediate and mart models belongs to [building clean DAGs](lesson:1.5).

Sources1

2.Declaring a source and resolving its real name

You declare sources in .yml files under a top-level sources: key. These files can sit at any depth inside the models/ directory and can have any name. Each source has a name, and inside it a list of tables. Three properties map the friendly names you use in code onto the object that actually exists in the warehouse:

- database names the database the raw data lives in, which is common on Snowflake. - schema defaults to the source's name, so you set it only when the two differ. - identifier names the physical table when it differs from the table's name.

sources:
  - name: jaffle_shop
    database: raw
    schema: postgres_backend_public_schema
    tables:
      - name: orders
        identifier: api_orders

Given that declaration, a model writes {{ source('jaffle_shop', 'orders') }} and dbt compiles it to raw.postgres_backend_public_schema.api_orders. The first argument is the source name and the second is the table name, never the physical identifiers. When you check a raw dependency, the compiled SQL is where you confirm that the declared names resolve to the object you expect. Reading compiled code in general is covered in [troubleshooting with compiled code](lesson:3.2).

Two details trip people up when they check that a raw object resolves correctly. First, dbt does not quote the database, schema or identifier by default. If a raw table was created with a case-sensitive name, set quoting (for example identifier: true) on the source or on a single table, and the table-level setting overrides the source-level one. Second, source properties are special. They live only in these .yml files, and you cannot set them from dbt_project.yml or a config() block.

select * from {{  ? ('jaffle_shop', 'orders') }}

Sources2

3.source() versus ref(): two kinds of DAG edge

The two functions split the DAG along one line: who created the object. source() points at data that arrived from outside dbt. ref() points at a node dbt builds itself, which means a model, a seed or a snapshot. Both return a Relation, both compile to the full object name, and both create a dependency that dbt uses for documentation and node selection. They differ in what sits at the other end of the edge.

ref() also does two jobs that a raw reference never needs. It fills in the schema for whatever environment you are working in, so a model in your dev target reads the dev copy of its upstream model and not production. It also uses these references to build the dependency graph, which is how dbt run knows to build stg_orders before fct_orders. Point ref() at a raw table and it fails to resolve, because no node by that name exists. Hard-code a model's name instead of using ref() and you lose both the build order and the environment switch.

Sources341

4.Verifying raw dependencies before you build

Declared edges only help if you check them. Once every raw read goes through source(), the DAG can answer two questions: which raw objects does this model depend on, and which models break if this raw table changes? Node selection answers both. A + in front of a resource selects it plus all its ancestors. A + after it selects it plus all its descendants. The source: method selects by source name, or by source and table.

$ dbt run --select source:jaffle_shop.orders+

Use dbt ls (alias dbt list) to look before you run anything. It takes the same selector arguments as dbt run but never connects to the database or runs a query. That makes it a cheap way to confirm that a model's lineage contains the sources you expect. Use --resource-type source to keep only sources in the output, and --output json when you need node properties. Two limits apply. Disabled models are left out of the results. And because dbt ls reads only the project, it checks the declared edges, not whether the raw table actually exists or holds good data.

Checking the data itself is a test's job. Sources accept the same data tests and descriptions as models, and you can run tests on them alone: dbt test --select "source:*" for every source, source:jaffle_shop for one source, or source:jaffle_shop.orders for a single table. Which assertions are worth making about raw data is covered in [testing assumptions for models and sources](lesson:5.2). Whether the raw data is arriving on time is a separate check, covered in [source freshness](lesson:6.2).

Sources5267

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Hard-coding raw.jaffle_shop.orders in a model works exactly like calling source(), as long as the query runs.Why is that wrong?

   A direct relation reference creates no DAG edge. The model drops out of lineage, out of source: selection and away from the tests on that source, and every model that hard-codes the name has to be edited when the raw table changes.

   Covered in Why raw objects need declaring

2. 2.ref() can point at any table in the warehouse, including raw loaded data.Why is that wrong?

   ref() resolves only to nodes dbt builds (models, seeds and snapshots). Raw data from Extract and Load tools has to be declared as a source and read with source().

   Covered in source() versus ref(): two kinds of DAG edge

3. 3.A source's database or schema can be set project-wide in dbt_project.yml or in a model's config() block.Why is that wrong?

   Source properties are special. They are declared only in .yml files under the models/ directory.

   Covered in Declaring a source and resolving its real name

4. 4.Running dbt ls confirms that the raw tables behind a model exist in the warehouse.Why is that wrong?

   dbt ls reads the project and resolves selectors, but it never queries the database. It checks the declared edges, not the physical objects.

   Covered in Verifying raw dependencies before you build