Lakeflow Connect Managed vs Standard Connectors and Their Components

1.Managed or standard: picking the ingestion layer

Lakeflow Connect offers ingestion at more than one level of the ETL stack. At the top are fully-managed connectors: ready-made connectors for popular enterprise applications such as Salesforce and databases such as SQL Server. Below them are standard connectors, which you build with Lakeflow pipelines or Structured Streaming. The docs describe standard connectors as ones that "trade some automation for broader source support and customization".

The two layers are related. Fully-managed connectors are built on top of Lakeflow pipelines. They add source-specific authentication, change data capture (CDC), edge-case handling, long-term API maintenance, automated retries and automated schema evolution. Lakeflow pipelines, in turn, are a declarative framework built on Structured Streaming. They handle orchestration, monitoring, data quality and errors once you define your transformations.

The guidance is to start at the most managed layer. If it doesn't meet your requirements (most often because it doesn't support your source), you drop down one layer. Sources with no managed connector at all can use community connectors (open source, built by the community, not backed by Databricks SLAs) or a custom connector that you build and run in your own workspace.

"Fully managed" sets a firm limit. With a managed connector you still choose the objects to ingest, the destination, the schedule, permissions and notifications. You cannot change how the ingestion itself runs. If you need to change that, move to Lakeflow pipelines or Structured Streaming. Auto Loader also sits outside the managed tier. It incrementally ingests files from S3, ADLS and GCS and works with Structured Streaming and Lakeflow pipelines, but it is not a fully-managed ingestion pipeline. Auto Loader is covered in its own lesson.

Sources123

2.Connector types by source

Once you know you want a managed connector, the next question is which kind. Lakeflow Connect groups its connectors by the type of source, and each type ingests in a different way:

- Database connectors (CDC) read relational databases such as MySQL, PostgreSQL, Oracle and SQL Server using change data capture. SQL Server also supports full snapshots. - Query-based connectors query the source database directly, without any CDC configuration. - SaaS connectors ingest from enterprise applications such as Salesforce, HubSpot, Jira and Workday. Each handles source-specific authentication, incremental reads, schema evolution and automated retries. - File source connectors ingest structured and unstructured files from enterprise file services such as Google Drive and SharePoint. - Streaming connectors continuously ingest from message buses and event streaming sources such as RabbitMQ.

The catalogue is large and keeps growing, and connectors are in various release states. Check the connector's own page before assuming it is generally available.

Sources45

3.Anatomy of a managed connector

Every managed connector is assembled from a few named components, and the connector type decides which ones it has. SaaS and file connectors are the simplest: a connection, an ingestion pipeline and destination tables. Query-based and streaming connectors have the same three parts. A query-based connector runs its queries against the source on a schedule, while a streaming connector reads messages continuously.

Database connectors add two more parts, because CDC must not miss changes. The ingestion gateway pulls snapshots, change logs and metadata from the source. It runs on classic compute, and it runs continuously so it captures changes before the source truncates its change logs. The gateway writes into staging storage, a Unity Catalog volume that holds extracted data until the ingestion pipeline applies it. Because changes wait in staging, the serverless ingestion pipeline can run on any schedule you like, and staging also helps with failure recovery. Staged data is purged automatically after 30 days.

The always-on gateway has a cost: you provision and pay for its classic compute even while the ingestion pipeline is idle.

Sources24

4.Where Unity Catalog governs the ingestion

Unity Catalog governs a managed ingestion pipeline from the credentials through to the output tables. The connection is a Unity Catalog securable object that holds the source's authentication details. For streaming sources it holds the endpoint as well, so the connector can authenticate without credentials in the pipeline configuration. A database connector's staging storage is a Unity Catalog volume. The destination tables are streaming tables: Delta tables with extra support for incremental data processing, registered in the catalog and schema you choose.

Lakeflow Connect also uses other platform features: Unity Catalog for governance, Lakeflow Jobs for orchestration, and monitoring across your pipelines. You can add governance controls to the ingested tables after they land. Managed pipelines' streaming tables accept a limited set of ALTER statements for column comments, column masks, row filters and tags:

ALTER MATERIALIZED VIEW view_name | ALTER STREAMING TABLE table_name
{
    ALTER COLUMN column_clause |
    SET ROW FILTER clause |
    DROP ROW FILTER |
    SET TAGS clause |
    UNSET TAGS clause
}

Sources21

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.A managed connector can be tuned to change how it reads from the source.Why is that wrong?

   Managed connectors let you choose objects, destination, schedule, permissions and notifications, but not the ingestion process. Use Lakeflow pipelines or Structured Streaming when you need that control.

   Covered in Managed or standard: picking the ingestion layer

2. 2.Every managed connector has an ingestion gateway and staging storage.Why is that wrong?

   Only database (CDC) connectors add a gateway and staging volume. SaaS and file connectors use just a connection, an ingestion pipeline and destination tables, and query-based connectors have no gateway or staging.

   Covered in Anatomy of a managed connector