Monitor Jobs and Pipelines with the Databricks CLI, REST API and SDK

1.One API, three ways in

In the Lakeflow Jobs UI you can browse the run history of each job and open the details of any single run. That stops working once you need to check hundreds of runs, feed results into a script, or ask the same question every morning. For that, the Monitor Lakeflow Jobs page points you to the Databricks CLI and the Jobs API. All the programmatic options sit on one layer: the Databricks REST API. The CLI wraps it, and so do the SDKs for Python, Java, Go and R. Most CLI commands map straight to a REST request. For example, databricks jobs get maps to GET /api/2.2/jobs/get.

Make sure you are on the current CLI. Versions 0.205 and above are the current Databricks CLI (1.0.0 and above is generally available). Versions 0.18 and below are the legacy CLI, and Databricks plans no support or new feature work for it. To see the options for any jobs command, add -h, for example databricks jobs <command-name> -h.

Sources123

2.Finding and inspecting job runs

Monitoring a job means asking questions about its runs. databricks jobs list-runs answers "which runs happened?" and returns them newest first, in descending order of start time. Its flags narrow the list down, so you aren't sorting through everything by hand.

Once you have a run ID, get-run returns that run's metadata. Two optional flags add more: --include-history adds the repair history, and --include-resolved-values adds the resolved parameter values. Large runs come back in pages. Arrays such as tasks, job_clusters, job_parameters and repair_history return only their first 100 elements. Pass the response's next_page_token as --page-token to fetch the rest.

databricks jobs get-run RUN_ID [flags]

Metadata tells you how a run went. It doesn't tell you what the run produced. For that, get-run-output retrieves the output and metadata of a single task run, including any value a notebook returned through dbutils.notebook.exit(). It has two limits worth memorising. The API returns only the first 5 MB of output, so store larger results in cloud storage. And runs are automatically removed after 60 days, so save any run results you need to keep before then.

Monitoring often leads straight to action, and the same command group handles that too. repair-run re-runs one or more tasks as part of the original job run, using the current job and task settings. The run must not be in progress. --rerun-all-failed-tasks repairs every failed task, and --rerun-dependent-tasks also re-runs downstream tasks that had succeeded. cancel-run is asynchronous: the run may still be running when the request completes. Its counterpart for a whole job is cancel-all-runs.

Sources45

3.Calling the REST API directly: paging and rate limits

When you skip the wrappers, every request needs the same parts: the workspace URL (account APIs use the account console URL and account ID instead), the operation type such as GET or POST, the operation path, authentication such as a Databricks OAuth token, and any payload or query parameters. Jobs and clusters are workspace resources, so you call them with your workspace URL as the host.

curl -X GET "$DATABRICKS_HOST/api/2.0/clusters/list" \
  -H "Authorization: Bearer $DATABRICKS_TOKEN"

A monitoring script that reads only the first page of results will quietly miss runs. LIST APIs return pages, not one large response, and the documented loop is the same for every LIST endpoint. Only the endpoint and the name of the array field in the response change.

    tables = []
    while True:
        response = requests.get(endpoint, headers=headers, params=params)
        response.raise_for_status()
        body = response.json()

        tables.extend(body.get("tables", []))

        # Stop when the response no longer includes a page token.
        page_token = body.get("next_page_token")
        if not page_token:
            break
        params["page_token"] = page_token

Polling job status in a tight loop can hit rate limits, which Databricks applies per endpoint and per workspace. A request over the limit returns HTTP 429 Too Many Requests. Retry with exponential backoff: double the wait after each 429, and cap both the wait time and the number of retries. Add jitter so that many clients don't all retry at the same moment. If the response includes a Retry-After header, wait at least that long.

Sources1

4.Monitoring pipelines: updates and events

A pipeline doesn't have job runs. It has updates, and each update produces events. The pipeline event log records audit logs, data quality checks, pipeline progress and data lineage. You can read it in the monitoring UI, by querying the event log directly, or through the Pipelines REST API, which the CLI wraps.

The pipelines command group has two kinds of commands. Project commands such as logs and history identify a pipeline by KEY, its unique name in the project's YAML file. KEY is optional if the project contains only one pipeline. Object commands such as list-updates, get-update and list-pipeline-events take a PIPELINE_ID. Time filters differ between the two groups. pipelines logs and pipelines history take timestamp strings such as 2025-01-15T10:30:00Z, while jobs list-runs takes integer start-time values.

databricks pipelines logs pipeline-name --update-id update-1 -n 10

databricks pipelines logs pipeline-name  ?  ERROR,METRICS --event-type update_progress --start-time 2025-01-15T10:30:00Z

Sources67

5.Scripting it in Python: the Databricks SDK

When a monitoring task grows into an application, the Databricks SDK for Python is the recommended route. It is in Beta and is okay to use in production. During the Beta, pin the specific minor version your code depends on. You start by creating a WorkspaceClient. With no arguments, it uses the Databricks unified authentication standard: it reads a configuration profile named by DATABRICKS_CONFIG_PROFILE, or the environment variables for your authentication type. You can hard-code host and token, but that is not recommended because it risks exposing secrets such as personal access tokens.

Inside a notebook it's simpler. The SDK is installed by default on clusters running Databricks Runtime 13.3 LTS or above. On older runtimes you install it, or upgrade it, with %pip. Default notebook authentication uses a temporary personal access token that Databricks creates in the background and deletes after the notebook stops. It has three limits: it works only on the driver node, not with configuration profiles, and not with Databricks Container Services. To call account-level APIs, use OAuth M2M, OAuth U2M or personal access token authentication instead.

%pip install databricks-sdk --upgrade

Sources8

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.The CLI and the REST API are separate surfaces, so some job information is only available through one of them.Why is that wrong?

   The CLI wraps the REST API, and most commands map to a REST request. For example, databricks jobs get corresponds to GET /api/2.2/jobs/get.

   Covered in One API, three ways in

2. 2.A single get-run call always returns every task of a large multi-task run.Why is that wrong?

   Array properties such as tasks return only their first 100 elements. You must follow next_page_token with --page-token to get the rest.

   Covered in Finding and inspecting job runs

3. 3.get-run-output returns the full output of a task, however large.Why is that wrong?

   The API returns only the first 5 MB of output. Larger results should be stored in a cloud storage service.

   Covered in Finding and inspecting job runs

4. 4.databricks pipelines logs with no filters shows the pipeline's entire event history.Why is that wrong?

   Without --update-id, it shows only the events of the most recent update.

   Covered in Monitoring pipelines: updates and events