Foundation Model APIs: Serving Modes and Provisioned Throughput Endpoints

1.What Foundation Model APIs give an LLM application

Every LLM application needs a model it can call. On Databricks the shortest path is Foundation Model APIs, a feature of Model Serving that lets you "access and query state-of-the-art open models from a serving endpoint." Databricks hosts the models, so your work is the application code and not the model infrastructure.

The documentation lists what this is for: checking whether a project is viable before you invest more, building a quick proof of concept, building a RAG chatbot by pairing a foundation model with a vector index, replacing proprietary models with open ones, comparing LLMs or swapping in a better one, and running production apps on "a scalable, SLA-backed LLM serving solution." The models are also a Databricks Designated Service, which means Databricks Geos manage data residency when customer content is processed.

Sources1

2.Picking a serving mode

The model is the same in every mode. What changes is how you pay for capacity and what performance you get. Pay-per-token is the starting point: Databricks preconfigures endpoints for the pay-per-token models, and you find them by clicking AI Gateway in the sidebar, where they are listed as system-provided model services.

For latency-sensitive production traffic on pay-per-token there is priority mode. You set the service_tier request parameter to "priority", and Databricks admits the request ahead of standard best-effort pay-per-token traffic. You opt in per request, there is no commitment, and you pay a higher per-token rate.

Provisioned throughput gives you dedicated capacity. It is the recommended mode for production, and it is required when you serve fine-tuned or fully custom weights. It comes in two options: on-demand, with no term, and reserved, with a fixed 1- or 3-month term at a lower per-unit rate. Provisioned throughput endpoints are also available with compliance certifications such as HIPAA. The last mode, AI Functions optimized models, is meant for batch inference.

Sources1

3.Requirements and model task types

Before an application can call Foundation Model APIs, three things must be in place. You need a Databricks API token to authenticate requests. You need serverless compute if you use provisioned throughput models. Your workspace must be in a supported region, and the pay-per-token and provisioned throughput modes each have their own list of regions.

The models fall into task types, and the task type tells you which model a feature needs. General purpose (chat) models handle multi-turn conversation, for example virtual assistants and support bots. Embedding models turn data into vectors for semantic search and RAG; the Databricks-hosted ones are databricks-qwen3-embedding-0-6b, databricks-gte-large-en and databricks-bge-large-en. Vision models analyse images and documents, and reasoning models cover code generation and agent orchestration. A RAG application usually needs two of these at once: an embedding model for retrieval and a chat model for generation.

Sources1

4.Creating a provisioned throughput endpoint

Databricks recommends serving the foundation models that come pre-installed in Unity Catalog, in the system catalog under the ai schema (system.ai). In the UI you open system.ai in Catalog Explorer, click the model, then click Serve this model. That opens the Create serving endpoint page. In the Up to dropdown you set the maximum tokens per second. The endpoint scales automatically, and Modify shows the minimum it can scale down to. One restriction: for Meta Llama models you must choose an Instruct version, because base versions cannot be deployed from Unity Catalog.

With the REST API, the request body depends on how the model measures capacity: in throughput bands or in model units. In both cases you first call the model optimization information API, GET api/2.0/serving-endpoints/get-model-optimization-info/{registered_model_name}/{version}. Its response says whether the model is optimizable and gives the chunk size, which is the increment you provision in.

{
  "optimizable": true,
  "model_type": "llama",
  "throughput_chunk_size": 980
}

# Send the POST request to create the serving endpoint
data = {
  "name": endpoint_name,
  "config": {
    "served_entities": [
      {
        "entity_name": model_name,
        "entity_version": model_version,
        "provisioned_model_units": desired_model_units,
      }
    ]
  },
}

response = requests.post(
  url=f"{API_ROOT}/api/2.0/serving-endpoints/pt", json=data, headers=headers
)

chunk_size = optimizable_info[' ? ']

# Desired provisioned throughput
desired_model_units = 2 * chunk_size

Sources2

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Provisioned throughput is the easiest way to start using Foundation Model APIs.Why is that wrong?

   Pay-per-token is the recommended starting point, and its endpoints are already preconfigured in the workspace. Provisioned throughput is the recommendation for production workloads that need guarantees.

   Covered in Picking a serving mode

2. 2.Every provisioned throughput endpoint is created by POSTing min/max throughput to /api/2.0/serving-endpoints.Why is that wrong?

   Only throughput-band models use min_provisioned_throughput and max_provisioned_throughput. Model-unit models use provisioned_model_units and POST to the /pt path.

   Covered in Creating a provisioned throughput endpoint

3. 3.Any Meta Llama variant in system.ai can be served on provisioned throughput.Why is that wrong?

   Only Instruct versions can be deployed from Unity Catalog. Base versions are not supported.

   Covered in Creating a provisioned throughput endpoint