What you will be able to do
- Choose between a managed connector and the three custom API-ingestion patterns based on volume and refresh needs
- Write a notebook that reads an API token from a secret and lands raw JSON responses in a Unity Catalog volume
- Run that notebook as a Lakeflow Jobs notebook task on a scheduled trigger, and know the scheduler's limits
1.Choosing how to ingest from an API
Ingesting from an API means pulling data over HTTP from a web service, usually as paginated JSON. Files and message buses have built-in sources, but there is no generic API source. Your code has to handle authentication, pagination and rate limits itself. So before you write any, check whether Lakeflow Connect already has a managed connector for the source. Salesforce, Workday, ServiceNow and Google Analytics have one, and a managed connector handles authentication, pagination and incremental extraction for you.
| Pattern | Use when | Where the data lands |
|---|---|---|
| Periodic pulls as a materialized view | Small-to-medium, bounded payloads such as reference data or daily FX rates | A materialized view, fully re-pulled on each pipeline update |
| Python Data Source API | High-volume or streaming APIs polled incrementally | A streaming table with checkpointed offsets |
| Decoupled ingestion with Auto Loader | You want API quirks kept out of transformation logic | Raw files in a Unity Catalog volume, written by a scheduled job |
The first two patterns run inside a pipeline. Every pipeline output is a Delta table governed by Unity Catalog, so API data reaches a governed table directly. The materialized-view pattern pulls the whole response again on every update, so use it only when the payload is bounded. The Python Data Source pattern checkpoints its progress, so a restart picks up from the last offset. The third pattern is the notebook-and-job pattern this lesson is about.
Checkpoint 1 of 6· Check yourself
An events API returns millions of new records a day, and a restart must not re-read everything. Which pattern fits?
A custom streaming source tracks offsets in checkpoints, so it reads incrementally. A materialized view re-reads the whole response on every update.
“so a restart resumes from the last offset instead of pulling the whole API again.”Source: docs.databricks.com
2.A notebook that lands API responses in a volume
The landing notebook does three things. It gets the API token from a Databricks secret scope at runtime; never hardcode it. It calls the endpoint with requests.get, passing the token in an Authorization: Bearer header and setting a timeout. Then it calls raise_for_status() and writes the JSON payload to a path under /Volumes/.... Landing in a Unity Catalog volume rather than an ungoverned bucket means lineage and access control reach back to the raw files.
Checkpoint 2 of 6· Fill the gap
Which dbutils utility returns the API token at runtime?
token = dbutils. ? .get(scope="<scope-name>", key="<secret-name>")Credentials come from a secret scope through dbutils.secrets. Widgets carry job parameters, not secrets.
Source: docs.databricks.comwith open(f"{volume_path}/events_{int(time.time())}.json", "w") as f: json.dump(resp.json()["data"], f)Each file is a separate unit for the downstream file ingestion to track. Because one file is written per run, no landed response is lost or ingested twice.
Real APIs paginate and throttle. Loop over the pages by following the API's next-page cursor until it runs out. Wrap each request in retry and backoff logic, so that one transient failure doesn't fail the whole run.
Checkpoint 3 of 6· Exam question
A notebook currently reads an entire 50-column, 80 million-row `transactions` table over JDBC into a DataFrame, then immediately filters it in Spark to keep only rows from the last 24 hours and only 4 of the columns before writing the result to a Unity Catalog table. Networking between Databricks and the source database is the bottleneck. Which change reduces the amount of data pulled over the JDBC connection itself?
Correct answer: A — Replace the `dbtable` option with a parenthesized subquery that selects only the four needed columns and filters `transaction_ts` to the last 24 hours, so the source database performs the filtering before crossing the network.
- A. Passing a parenthesized subquery as `dbtable` makes the source database itself execute the column selection and date filter, so only the reduced result set is transferred over the JDBC connection instead of all 50 columns and every historical row.
- B. JDBC pushdown of column and predicate pruning is not guaranteed for arbitrary sources, and relying on it after `.load()` still risks pulling the full table across the network if pushdown does not occur for this configuration.
- C. Adding partitions changes how the same full scan is split across connections; it does not reduce the total number of columns or rows read from the source, so total network transfer stays the same.
- D. A smaller `fetchSize` only changes batching per round trip for the same full result set; it does not reduce the number of columns or rows pulled from the source database.
3.Orchestrating the notebook with Lakeflow Jobs
Lakeflow Jobs is the orchestration layer. A job is made of tasks, and a task can run a notebook, a pipeline, a managed connector or a SQL query, among others. To run the landing notebook, add a Notebook task. Its source is either the workspace or a Git provider; Databricks recommends a remote Git repository for versioning. Choose compute that can run the notebook's code. On serverless compute, install libraries such as requests inside the notebook with %pip install or the Environment pane. You can pass task parameters as key-value pairs and read them in the notebook with dbutils.widgets.
Checkpoint 4 of 6· Check yourself
The landing notebook is written in Python. Which compute can its notebook task use?
A notebook task can run on a SQL warehouse only if the notebook is written entirely in SQL, with SQL as its default language.
“You can select a SQL warehouse as the compute for a notebook task only when the notebook is written entirely in SQL”Source: docs.databricks.com
A Scheduled trigger runs the job on a timetable. A Simple schedule repeats at an interval, for example every twelve hours, and the scheduler picks the time of the first run. An Advanced schedule lets you set the period, start time and time zone, and you can edit it in Quartz cron syntax. Runs triggered by a schedule are at least 10 seconds apart. Under a daylight-saving time zone, hourly runs can be skipped or delayed, so choose UTC for strict hourly runs. The scheduler isn't built for low latency, and runs can occasionally start several minutes late. One more limit: if the notebook's total cell output exceeds 30MB, or any single cell's output exceeds 8MB, the run is cancelled and marked as failed. Don't print whole API payloads.
Checkpoint 5 of 6· Put it in order
Put the steps for adding a schedule to an existing job in order
- 1.Set Trigger type to Scheduled
- 2.Open the job from Jobs & Pipelines
- 3.In the Job details pane, click Add trigger
- 4.Click Save
- 5.Choose Simple or Advanced and set the schedule
The trigger is added from the job's details pane: pick Scheduled, set the schedule, then save.
“In the Job details pane, click Add trigger.”Source: docs.databricks.com
Checkpoint 6 of 6· Exam question
During a code review, a data engineer notices that a coworker's notebook task builds a JDBC connection string that includes the literal database username and password directly in the `.option("password", ...)` call, and this notebook is scheduled to run daily as part of a Lakeflow Job that several other engineers can view. What is the recommended way to fix this before the job is promoted to production?
Correct answer: A — Store the username and password in a Databricks secret scope and retrieve them at runtime with `dbutils.secrets.get()`, so the plaintext values never appear in the notebook source or job configuration.
- A. Secret scopes are the recommended mechanism for JDBC credentials: values retrieved with `dbutils.secrets.get()` are redacted in notebook output and never stored as plaintext in the source or job configuration.
- B. Ordinary Lakeflow Job task parameters are not encrypted or hidden by default, so putting a password in a plain parameter still exposes it in the job's configuration to anyone who can view the job.
- C. Base64 is a reversible encoding, not encryption, so anyone who can view the notebook can trivially decode the value; it provides no real credential protection.
- D. The engineers who can already run the scheduled job can still see the notebook contents, so folder-level restriction to admins does not stop the people who need to operate the job from seeing the plaintext password.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Any notebook task can run on a SQL warehouse to save cost.Why is that wrong?
A notebook task can use a SQL warehouse only if the notebook is written entirely in SQL. A Python API-landing notebook needs a cluster or other supported compute.
2.A Simple schedule lets you choose exactly when the first run happens.Why is that wrong?
With a Simple schedule, the scheduler picks the first run time. If you need to control the start time, use an Advanced schedule.
3.The materialized-view pull pattern is fine for any API, because the pipeline handles incremental reads.Why is that wrong?
A materialized view re-reads the full API response on every update, so it suits only bounded payloads.
Covered in Choosing how to ingest from an API
Practise it for real
Land raw API responses in a Unity Catalog volume on a schedule, using a notebook run by Lakeflow Jobs
1.Store the API token in a Databricks secret scope
Why: Credentials must never be hardcoded in source code
You should see: dbutils.secrets.get with your scope and key returns the token inside a notebook
2.Write a notebook that calls the API with requests.get and a Bearer header, calls raise_for_status(), and writes the JSON data to a timestamped file under /Volumes/<catalog>/<schema>/<volume>/
Why: Landing in a volume keeps the raw files under Unity Catalog governance
You should see: Each run adds one new events_<timestamp>.json file to the volume
3.Create a job with a Notebook task that points to the notebook and runs on serverless compute, installing requests with %pip install if needed
Why: A Python notebook can't run on a SQL warehouse, and serverless compute installs libraries inside the notebook
You should see: Run now completes and writes a file
4.Add a Scheduled trigger with a Simple schedule, for example every 12 hours, and save
Why: Lakeflow Jobs provides the recurring orchestration
You should see: The job lists the trigger and shows the next scheduled run
Stuck? Get a nudge
If the run fails with no error in your code, check whether a cell printed the whole payload. Output over 8MB in one cell cancels the run.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/ldp/api-ingestionOfficial docs
“there's no built-in generic API source, so you handle authentication, pagination, and rate limits yourself.”
↩︎ Choosing how to ingest from an API“Before you write any custom API-ingestion code, check whether a managed connector already exists for your source.”
↩︎ Choosing how to ingest from an API“Never hardcode credentials in pipeline source code.”
↩︎ A notebook that lands API responses in a volume“Add retry and backoff logic around the request for resilience.”
↩︎ A notebook that lands API responses in a volume“Schedule the notebook or script to run on its own with Lakeflow Jobs.”
↩︎ Orchestrating the notebook with Lakeflow Jobs“This pattern re-reads the full API response on every pipeline update, so use it only when the payload is bounded.”
↩︎ Exam trap 3“A scheduled job lands the raw API responses as files in a Unity Catalog volume, and the pipeline picks them up with Auto Loader.”
↩︎ Prediction“so a restart resumes from the last offset instead of pulling the whole API again.”
↩︎ Checkpoint - 2.https://docs.databricks.com/aws/en/ldp/loadOfficial docs
“Every streaming table and materialized view a pipeline creates is stored as a Delta table by default”
↩︎ Choosing how to ingest from an API“land data in a Unity Catalog volume rather than an ungoverned bucket path, so lineage and access control extend back to the landing zone”
↩︎ A notebook that lands API responses in a volume - 3.https://docs.databricks.com/aws/en/data-engineeringOfficial docs
“A job can consist of one or more tasks that run notebooks, pipelines, managed connectors, SQL queries”
↩︎ Orchestrating the notebook with Lakeflow Jobs - 4.
“If you use Serverless compute, install libraries directly inside notebook, using the Environment pane or using %pip install.”
↩︎ Orchestrating the notebook with Lakeflow Jobs“the run is canceled and marked as failed.”
↩︎ Orchestrating the notebook with Lakeflow Jobs“You can select a SQL warehouse as the compute for a notebook task only when the notebook is written entirely in SQL”
↩︎ Exam trap 1“You can select a SQL warehouse as the compute for a notebook task only when the notebook is written entirely in SQL”
↩︎ Checkpoint - 5.https://docs.databricks.com/aws/en/jobs/scheduledOfficial docs
“Databricks enforces a minimum interval of 10 seconds between subsequent runs triggered by the schedule of a job”
↩︎ Orchestrating the notebook with Lakeflow Jobs“The job scheduler is not intended for low-latency jobs.”
↩︎ Orchestrating the notebook with Lakeflow Jobs“You cannot specify the time for the first run for a simple schedule.”
↩︎ Exam trap 2“In the Job details pane, click Add trigger.”
↩︎ Checkpoint