Querying Foundation Model APIs from an LLM Application

1.Ways an application can call a foundation model

After you pick a serving mode, the application needs a way to send requests. Model Serving uses one OpenAI-compatible API for Databricks-hosted Foundation Model APIs and for external models. That lets the same calling code work whichever provider is behind the endpoint. All of these requests go through Unity Gateway, where you can apply rate limits, budgets and guardrails.

If you just want to call an LLM, the docs suggest model services in Unity Gateway. These give you governed access to Databricks-hosted models without configuring a serving endpoint. If you serve the model on an endpoint you host yourself, the calling code is the same except for one value: you replace the model service name (for example system.ai.claude-sonnet-4-5) with your endpoint name.

Sources1

2.Authenticating the application

The OpenAI client, the REST API and the MLflow Deployments SDK all need a Databricks API token to send scoring requests. Which token depends on the stage. For testing and development, use a personal access token that belongs to a service principal rather than to a workspace user. In production, switch to machine-to-machine OAuth tokens.

The databricks-openai package means you don't have to wire up credentials by hand: it "provides an OpenAI client with authorization automatically configured." In a notebook you install it and then run dbutils.library.restartPython(). You only use the plain openai client, with your token as api_key and the workspace's /ai-gateway/mlflow/v1 URL as base_url, when you query foundation models from outside the workspace.

Sources1

3.Sending a chat request and reading the response

Inside the workspace, a chat call is a standard OpenAI chat.completions.create call. Only the model value is Databricks-specific.

from databricks_openai import DatabricksOpenAI

client = DatabricksOpenAI()

response = client.chat.completions.create(
    model="system.ai.claude-sonnet-4-5",
    messages=[
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "What is a mixture of experts model?",
      }
    ],
    max_tokens=256
)

A REST call carries the same fields in a JSON body: messages, max_tokens and temperature. The response follows the OpenAI chat completion shape, including a usage object that counts prompt and completion tokens. On pay-per-token, that token count is what you are billed for.

{
  "model": "databricks-claude-sonnet-4-5",
  "choices": [
    {
      "message": {},
      "index": 0,
      "finish_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 7,
    "completion_tokens": 74,
    "total_tokens": 81
  },
  "object": "chat.completion",
  "id": null,
  "created": 1698824353
}

Sources2

4.Batch inference from SQL with ai_query

Not every LLM application serves one user at a time. To transform rows in a table, you can call the same foundation models from SQL with the built-in ai_query function. This is the AI Functions path, and the serving mode recommended for batch inference is AI Functions optimized models. Note that the function is in Public Preview and its definition might change.

SELECT  ? (
    "system.ai.claude-sonnet-4-5",
    "Can you explain AI in ten words?"
  )

Sources21

5.Serving an agent app that calls Foundation Model APIs

A whole LLM application, such as an agent with a chat UI, can be deployed on Databricks Apps. Apps "gives you full control over the agent code, server configuration, and deployment workflow." The app still gets its model from a foundation model endpoint. The recommended way to wire this up is to send every LLM call through Unity Gateway: in the agent code, pass the gateway endpoint name as model and set use_ai_gateway=True on the Databricks LLM client (AsyncDatabricksOpenAI or ChatDatabricks in LangChain). The client then handles authentication. Because the gateway sits in the request path, you can swap models, attribute cost per app, and inspect or replay traffic without changing agent code or rotating provider credentials.

Sources3

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.A developer's personal access token is the right credential for an app querying Foundation Model APIs in production.Why is that wrong?

   Production should use machine-to-machine OAuth tokens. Even for development, the recommendation is service-principal PATs, not tokens tied to workspace users.

   Covered in Authenticating the application

2. 2.DatabricksOpenAI works the same way from outside the workspace.Why is that wrong?

   To query from outside the workspace you use the plain OpenAI client, configured with your token and the workspace URL.

   Covered in Authenticating the application

Practise it for real

Query a pay-per-token foundation model from a notebook, first with the OpenAI-compatible client and then from SQL

1. 1.In a notebook, run pip install -U databricks-openai, then dbutils.library.restartPython()

   Why: The package provides an OpenAI client with authorization configured for Databricks

   You should see: The package installs and the Python process restarts

2. 2.Run the DatabricksOpenAI chat.completions.create example with model="system.ai.claude-sonnet-4-5"

   Why: This is the recommended OpenAI-compatible path for calling a model service

   You should see: A chat completion object whose usage shows prompt, completion and total tokens

3. 3.Change the model argument to a different general purpose model from the supported list and run it again

   Why: Shows that the shared API lets you compare or swap models without changing any other code

   You should see: A response in the same format from the other model

4. 4.In a SQL cell, run SELECT ai_query("system.ai.claude-sonnet-4-5", "Can you explain AI in ten words?")

   Why: ai_query is the AI Functions path for model inference from SQL, used for batch workloads

   You should see: A single text response returned as the query result