What you will be able to do
- Explain the three layers of Databricks ingestion products and say which one Databricks recommends you start with
- Decide between a Lakeflow Connect managed connector and a standard connector for a given source
- Describe how managed connectors use Unity Catalog for governance and credentials
- Say when community connectors, custom connectors or Partner Connect ingestion partners fit
Key concept
Start with the most managed layer — Databricks ingestion products are arranged in layers that go from fully customizable to fully managed. Begin with the most managed option that supports your source and requirements, and move to a more customizable layer only when it cannot do what you need.
1.Three layers, and why you start at the most managed one
Lakeflow Connect is the name for all of Databricks' ingestion connectors. It covers files, enterprise applications, databases, message buses and more. Most ingestion questions on the exam come down to one decision: how much of the pipeline do you want to build and maintain yourself? Lakeflow Connect gives you two broad options. One is a fully-managed service with out-of-the-box connectors, which lets you build ingestion pipelines quickly and keeps long-term maintenance costs low. The other is a custom pipeline built with Lakeflow pipelines or Structured Streaming, for when you need more control.
The documentation lists the three layers of ingestion products, "ordered from most customizable to most managed". Each layer builds on the one below it. Lakeflow pipelines extend Structured Streaming with a declarative framework. Managed connectors build on Lakeflow pipelines and add source-specific features on top.
| Layer | What it gives you | What you still own |
|---|---|---|
| Structured Streaming | A streaming engine with end-to-end fault tolerance and exactly-once processing, using Spark APIs | Everything else around the stream |
| Lakeflow pipelines | A declarative framework that manages orchestration, monitoring, data quality and errors | Your transformations and the source-specific logic |
| Managed connectors | Everything in Lakeflow pipelines plus source-specific authentication, CDC, edge case handling, long-term API maintenance, automated retries and automated schema evolution | Very little, but only for the supported sources |
The decision rule is simple. Use the managed connector if one exists for your source and it meets your requirements. If it doesn't, for example because your source isn't supported, move down one layer to Lakeflow pipelines. Go all the way to Structured Streaming only when you need full customization.
Checkpoint 1 of 4· Put it in order
Put these layers in the order you try them, following Databricks' recommendation
- 1.Structured Streaming
- 2.Lakeflow pipelines
- 3.Managed connector
You start at the most managed layer and move down one layer at a time, only when the current layer doesn't meet your requirements.
“If it doesn't satisfy your requirements (for example, if it doesn't support your data source), drop down to the next layer.”Source: docs.databricks.com
The layers are not the only ingestion methods. For files in cloud object storage, the standard route is Auto Loader or COPY INTO, and volume decides between them: COPY INTO suits files in the order of thousands over time, while Auto Loader is for millions or more. Lakeflow Connect can also ingest from local files. If your own application produces the data, direct write with the Zerobus Ingest API writes records straight into streaming tables, with no files or connector in between. And if a query-based connector fits a database source, its connection can be a Unity Catalog foreign catalog, which is how Lakehouse Federation is used for ingestion.
2.Managed versus standard connectors
Lakeflow Connect has two kinds of connectors. Managed connectors produce ingestion pipelines that are "governed by Unity Catalog and powered by serverless compute and Lakeflow pipelines". Standard connectors run inside your own Structured Streaming code, Lakeflow pipelines or Databricks SQL. The trade-off is stated plainly: you can use standard connectors, "which trade some automation for broader source support and customization." A managed connector does more of the work for you. A standard connector supports more sources and lets you control more of the pipeline.
Lakeflow Connect groups its connectors by the type of source. This grouping helps you map a scenario to the right family of connectors, and the last column shows what tips the choice.
| Connector type | Example sources | How it ingests and when to pick it |
|---|---|---|
| Database connectors (managed) | MySQL, PostgreSQL, SQL Server | Change data capture through an ingestion gateway and staging storage. Pick it when the source supports CDC and you want continuous change capture |
| Query-based connectors | Databases without CDC configured | Queries the source database directly on a schedule, with no CDC. Pick it when CDC isn't available or configured. The connection can be a direct Unity Catalog connection or a foreign catalog (Lakehouse Federation) |
| SaaS connectors (managed) | Salesforce, HubSpot, Jira, Workday | Calls the source application's APIs. Pick it for enterprise applications |
| File connectors | Managed: Google Drive, SharePoint. Standard: cloud object storage, SFTP servers | Structured and unstructured files. Pick managed for a configuration-based service, standard (Auto Loader or COPY INTO) for more control or wider file sources |
| Streaming connectors | Managed: RabbitMQ. Standard: Apache Kafka, Amazon Kinesis, Google Pub/Sub, Apache Pulsar | Continuous reads from message buses. Pick the layer by how much customization you need |
| Direct write (Zerobus Ingest API) | Your own application | Writes records directly to streaming tables. Pick it when your application produces the records |
Frequency is a separate choice from connector type. Ingestion pipelines can run on a recurring schedule or continuously: batch ingestion uses triggered mode, which processes new data on a schedule or when manually triggered, and streaming ingestion uses continuous mode, which processes new data as it arrives.
Different connector types are built from different parts. SaaS and file connectors use a connection, an ingestion pipeline and destination tables. Database connectors also include an ingestion gateway and staging storage to support continuous change capture. A query-based connector queries the source directly on a schedule, so it has no gateway and no staging storage. These differences affect how often each one can run. You can give an ingestion pipeline one or more schedules, and Lakeflow Connect automatically creates a job for each schedule. A database connector's gateway, by contrast, runs in its own job as a continuous task.
Managed connectors ingest incrementally. The first run loads all of the selected data and starts tracking changes in the source. Each later run uses that change tracking to load only what has changed, when the source allows it. How this works depends on the source. SQL Server supports both change tracking and CDC, while "the Salesforce connector selects a cursor column from a set list of options." Some sources and tables don't support incremental ingestion yet.
Checkpoint 2 of 4· Match them up
Match each connector component to its role
Tap a term, then the definition that fits it.
Every connector has a connection, a pipeline and destination tables. Only database connectors add a gateway and staging storage, because they need continuous change capture.
“Database connectors also include an ingestion gateway and staging storage to support continuous change capture.”Source: docs.databricks.com
Checkpoint 3 of 4· Exam question
A data engineering team must load Salesforce opportunity and account records into Unity Catalog-governed tables, keep transformation logic minimal, and avoid maintaining custom authentication or change-data-capture code. Which ingestion approach best fits these requirements?
Correct answer: A — Use a Lakeflow Connect managed connector for Salesforce, which handles authentication, change data capture, and schema evolution automatically under Unity Catalog governance.
- A. A managed connector for Salesforce automates authentication, change data capture, and schema evolution while producing pipelines governed by Unity Catalog, matching the requirement for minimal engineering effort.
- B. A standard connector still requires the team to build and maintain streaming and transformation logic themselves, which is more effort than needed when a fully managed option already exists for Salesforce.
- C. Auto Loader is built to incrementally ingest files landing in cloud object storage, not to pull records from a SaaS application's change feed, so it does not fit this source type.
- D. A custom JDBC or REST notebook forces the team to build and maintain authentication, pagination, and change-tracking logic that a managed connector already provides out of the box.
3.Unity Catalog governance, community connectors and ingestion partners
Governance is often the deciding requirement in a scenario. Lakeflow Connect uses core Databricks features: it "offers governance using Unity Catalog, orchestration using Lakeflow Jobs, and holistic monitoring across your pipelines." Credentials are a concrete example. In a managed streaming connector, the connection is a Unity Catalog securable object that stores the source endpoint and credentials, and the connector uses it to authenticate without requiring credentials in your pipeline configuration. So secrets are governed in Unity Catalog and stay out of your code.
What if a source has no managed connector? Lakeflow Connect offers two more options before you write a fully custom pipeline. Community connectors are open source and maintained by the community. They are "not backed by Databricks SLAs." Custom connectors let you build, test, deploy and run your own connector in your own workspace for a source that has no managed connector.
Partner connectors are the last option. Databricks validates technology partner integrations that offer "low-code, scalable data ingestion from a variety of sources into Databricks." Partner Connect, available from Marketplace in the sidebar, "offers the simplest way to connect your Databricks workspace to a data ingestion partner solution." When you connect, you select a catalog for the partner to write to. Partner Connect then creates a SQL warehouse, a service principal named <PARTNER>_USER with a personal access token, and Unity Catalog grants such as USE CATALOG and CREATE SCHEMA. So a partner authenticates with that personal access token, not with a Unity Catalog connection like a managed connector. Pay attention to the governance caveat. If a partner doesn't support Unity Catalog through Partner Connect, it writes to the workspace default catalog. If the workspace isn't Unity Catalog-enabled, it writes to hive_metastore.
Checkpoint 4 of 4· Check yourself
You connect an ingestion partner through Partner Connect and select a catalog, but the partner does not support Unity Catalog with Partner Connect. Your workspace is Unity Catalog-enabled. Where does the partner write?
When the partner doesn't support Unity Catalog with Partner Connect, Databricks falls back to the workspace default catalog. It uses hive_metastore only when the workspace isn't Unity Catalog-enabled.
“If a partner doesn't support Unity Catalog with Partner Connect, the workspace default catalog is used.”Source: docs.databricks.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Start with Structured Streaming, because the most customizable layer can handle any requirement.Why is that wrong?
Databricks recommends the opposite: start with the most managed layer and move down only when it doesn't meet your requirements.
Covered in Three layers, and why you start at the most managed one
2.Managed connectors support more sources than standard connectors because they are the premium option.Why is that wrong?
It's the other way round. Standard connectors give up some automation in exchange for broader source support and more customization.
Covered in Managed versus standard connectors
3.Partner Connect always writes partner data into hive_metastore.Why is that wrong?
You select a target catalog. Databricks falls back to the workspace default catalog if the partner lacks Unity Catalog support, and uses hive_metastore only when the workspace isn't Unity Catalog-enabled.
Covered in Unity Catalog governance, community connectors and ingestion partners
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/ingestionOfficial docs
“the three layers of ingestion products, ordered from most customizable to most managed”
↩︎ Three layers, and why you start at the most managed one“They extend Lakeflow pipelines functionality to also include source-specific authentication, CDC, edge case handling, long-term API maintenance, automated retries, automated schema evolution”
↩︎ Three layers, and why you start at the most managed one“Triggered: Processes new data on a schedule or when manually triggered.”
↩︎ Managed versus standard connectors“Continuous: Processes new data as it arrives in the source.”
↩︎ Managed versus standard connectors“Databricks recommends starting with the most managed layer.”
↩︎ Key concept“Databricks recommends starting with the most managed layer.”
↩︎ Exam trap 1“Databricks recommends starting with the most managed layer.”
↩︎ Prediction“If it doesn't satisfy your requirements (for example, if it doesn't support your data source), drop down to the next layer.”
↩︎ Checkpoint - 2.
“This allows you to quickly create robust ingestion pipelines while minimizing long-term maintenance costs.”
↩︎ Three layers, and why you start at the most managed one“Write records directly to streaming tables from your application using the Zerobus Ingest API.”
↩︎ Three layers, and why you start at the most managed one“Lakeflow Connect offers simple and efficient connectors to ingest data from local files, popular enterprise applications, databases, cloud storage, message buses, and more.”
↩︎ Three layers, and why you start at the most managed one“it offers governance using Unity Catalog, orchestration using Lakeflow Jobs, and holistic monitoring across your pipelines.”
↩︎ Unity Catalog governance, community connectors and ingestion partners“not backed by Databricks SLAs”
↩︎ Unity Catalog governance, community connectors and ingestion partners“Custom connectors let you build your own connector for a source that has no managed connector.”
↩︎ Unity Catalog governance, community connectors and ingestion partners - 3.
“If you're going to ingest files in the order of thousands over time, you can use COPY INTO.”
↩︎ Three layers, and why you start at the most managed one“If you are expecting files in the order of millions or more over time, use Auto Loader.”
↩︎ Three layers, and why you start at the most managed one“These integrations enable low-code, scalable data ingestion from a variety of sources into Databricks.”
↩︎ Unity Catalog governance, community connectors and ingestion partners - 4.
“Managed connectors produce ingestion pipelines that are governed by Unity Catalog and powered by serverless compute and Lakeflow pipelines”
↩︎ Managed versus standard connectors“Ingest data from relational databases including MySQL, PostgreSQL, and SQL Server using change data capture.”
↩︎ Managed versus standard connectors“A query-based connector queries the source database directly on a schedule, without a gateway or staging storage.”
↩︎ Managed versus standard connectors“Either a direct Unity Catalog connection (for foreign connection ingestion) or a Unity Catalog foreign catalog (for foreign catalog ingestion using Lakehouse Federation)”
↩︎ Managed versus standard connectors“For each schedule that you add to a pipeline, Lakeflow Connect automatically creates a job for it.”
↩︎ Managed versus standard connectors“it uses that change tracking to ingest only the data that's changed from the prior run”
↩︎ Managed versus standard connectors“The managed connector uses this connection to authenticate without requiring credentials in your pipeline configuration.”
↩︎ Unity Catalog governance, community connectors and ingestion partners“you can also use standard connectors, which trade some automation for broader source support and customization.”
↩︎ Exam trap 2“Database connectors also include an ingestion gateway and staging storage to support continuous change capture.”
↩︎ Checkpoint - 5.
“Cloud object storage Ingest files from S3, ADLS, or GCS using Auto Loader or COPY INTO.”
↩︎ Managed versus standard connectors - 6.
“Partner Connect offers the simplest way to connect your Databricks workspace to a data ingestion partner solution.”
↩︎ Unity Catalog governance, community connectors and ingestion partners“A Databricks personal access token that is associated with the <PARTNER>_USER service principal.”
↩︎ Unity Catalog governance, community connectors and ingestion partners“If your workspace isn't Unity Catalog-enabled, hive_metastore is used.”
↩︎ Exam trap 3“If a partner doesn't support Unity Catalog with Partner Connect, the workspace default catalog is used.”
↩︎ Checkpoint