What you will be able to do
- Package custom preprocessing and prediction logic as an mlflow.pyfunc.PythonModel that Model Serving can host
- Validate a logged model's dependencies and input with mlflow.models.predict before deployment
- Create a serving endpoint for a Unity Catalog model version through the Serving UI, the REST API or the MLflow Deployments SDK
- Choose a workload type, workload size, provisioned concurrency and scale-to-zero setting for a given traffic pattern
- Roll a live endpoint to a new model version, or split traffic with a challenger, without downtime
Key concept
Served entity — A served entity is one Unity Catalog model version in an endpoint's configuration, together with the compute it runs on: workload type, workload size or concurrency range, and the scale-to-zero setting. Creating, resizing, updating and traffic-splitting an endpoint all come down to editing its list of served entities.
1.What a Model Serving endpoint gives a custom model
A Model Serving endpoint turns a registered model into an API that other systems call over HTTP. Databricks uses the term custom model for anything you train or write yourself, as opposed to foundation models and external models. That covers a scikit-learn classifier, an XGBoost booster, a Hugging Face transformer, or plain Python with no ML library at all. All of these deploy the same way.
Every custom model takes the same path to an endpoint:
- Log it in MLflow format, using a built-in flavor such as
mlflow.sklearnor a custompyfuncwrapper. - Register it, preferably in Unity Catalog.
- Create a serving endpoint whose configuration points at one registered version.
The endpoint is a managed REST service that autoscales on serverless compute. The calling application sends a request and waits for the prediction in the response. Use this pattern when each event needs its own answer right away, such as a payment service that must score each transaction before approving it.
Batch and streaming scoring don't meet that requirement, even with the same registered model. A nightly job that loads the model with mlflow.pyfunc.load_model and scores a DataFrame produces results hours after each transaction. A pipeline flow writes its scores to a table, so the service that is waiting never gets a response. A notebook rerun by hand is not a service at all. Batch, real-time and streaming serving trade-offs compares the three patterns. For a per-request latency budget measured in milliseconds, the answer is an endpoint called synchronously.
Checkpoint 1 of 9· Put it in order
A team has trained a scikit-learn fraud classifier and needs a payment service to score each transaction over HTTP. Put the steps for getting the model there in order.
- 1.Create a Model Serving endpoint whose configuration points at one registered model version
- 2.Log the model in MLflow format with a built-in flavor such as mlflow.sklearn, or with a custom pyfunc wrapper
- 3.Have the payment service send each transaction to the endpoint's REST API and wait for the prediction
- 4.Register the logged model, preferably in Unity Catalog
Every custom model takes the same path. You log it in MLflow format, register it, then create an endpoint that serves one registered version. Only then can the application call the endpoint over REST for a prediction on each request.
“After the model is logged, register it in the Unity Catalog (recommended) or the workspace registry.”Source: docs.databricks.com
Checkpoint 2 of 9· Exam question
A team has trained a custom scikit-learn fraud-detection model and registered it in Unity Catalog. The application must return a prediction within 50 milliseconds of each transaction as it happens, and transactions arrive continuously throughout the day. Which approach should the team use to serve predictions?
Correct answer: A — Deploy the registered model to a Model Serving endpoint as a served entity, then have the transaction service call the endpoint's REST API synchronously for each incoming transaction.
- A. A Model Serving endpoint keeps the model loaded and ready to respond to individual REST calls with single-digit-to-double-digit millisecond overhead, which matches a per-transaction, sub-50ms latency requirement.
- B. A nightly batch job scores transactions long after they occur, so it cannot meet a requirement that each transaction be scored the moment it arrives.
- C. A streaming pipeline flow scores records as micro-batches land in a table, which adds pipeline trigger and table-commit latency that is far larger than the required per-transaction response time.
- D. Manually rerunning a notebook cell depends on an analyst being available and cannot deliver a consistent, automated response within a fixed millisecond budget for every transaction.
2.Packaging custom logic as a pyfunc model
Use mlflow.pyfunc.PythonModel when a built-in flavor can't express what the endpoint has to do. Typical cases are:
- inputs that need preprocessing before they reach the model
- a framework MLflow doesn't support natively
- raw outputs that need post-processing
- branching logic that depends on the request
- code that isn't a trained model at all
A PythonModel splits its work between two methods, and where you put each piece of work affects latency. load_context runs once when the model loads, so weights, tokenizers and lookup tables belong there. predict runs on every request and should contain only per-request work. If you load a large file inside predict, every request pays that cost again. Shared helper modules ship with the model through code_path and can be imported inside load_context.
class CustomModel(mlflow.pyfunc.PythonModel):
def load_context(self, context):
self.model = torch.load(context.artifacts["model-weights"])
from preprocessing_utils.my_custom_tokenizer import CustomTokenizer
self.tokenizer = CustomTokenizer(context.artifacts["tokenizer_cache"])
def format_inputs(self, model_input):
# insert some code that formats your inputs
pass
def format_outputs(self, outputs):
predictions = (torch.sigmoid(outputs)).data.numpy()
return predictions
def predict(self, context, model_input):
model_input = self.format_inputs(model_input)
outputs = self.model.predict(model_input)
return self.format_outputs(outputs)During deployment, Databricks builds a container from the logged model. Anything the model imports must be listed in the model, because the base image covers only some system-level packages. A signature, input example and pip_requirements are attached when you log the model, as covered in Model signature, input example and pip requirements. For serving, two of these matter most. Unity Catalog won't register a model without a signature, and the listed packages must install cleanly in the container. Built-in flavors capture their dependencies automatically. For a custom pyfunc you add them explicitly, with pip_requirements, extra_pip_requirements or conda_env.
Databricks Runtime ML has a specific trap here. It ships mlflow-skinny instead of the full mlflow package. If you log a pyfunc there without pip_requirements, mlflow-skinny is written to the model's conda.yaml, and Model Serving can't build the image from it. The fix is to pin mlflow==<version> yourself.
mlflow.pyfunc.log_model(
name="model",
python_model=your_model,
pip_requirements=["mlflow==3.8.1"], # use mlflow, not mlflow-skinny
registered_model_name="catalog.schema.model_name",
)Endpoint deployment is a slow way to find a missing package. Run mlflow.models.predict against the logged model instead. With the default env_manager="virtualenv", it rebuilds the model's dependencies in an isolated environment that simulates serving. A missing or conflicting package then fails in your notebook within minutes, rather than after a deployment that can take ten minutes or more. pip_requirements_override lets you try other package versions without logging the model again. Once the right versions are known, update_model_requirements writes them into the logged model in place.
mlflow.models.predict(
model_uri=model_uri,
input_data={"col1": 34.2, "col2": 11.2, "col3": "green"},
content_type="json",
env_manager="virtualenv",
install_mlflow=False,
pip_requirements_override=["pillow==10.3.0", "scipy==1.13.0"],
)Checkpoint 3 of 9· Check yourself
A data scientist on Databricks Runtime ML logs a custom pyfunc model without pip_requirements and registers it in Unity Catalog. Deploying it to a serving endpoint fails because the container image can't be built. What is the most likely cause, and what fixes it?
Databricks Runtime ML ships mlflow-skinny. A pyfunc logged there without pip_requirements records mlflow-skinny in conda.yaml, and Model Serving can't build its image from that. Pinning the full mlflow package fixes it. The signature option doesn't fit, because Unity Catalog wouldn't have registered the model without one.
“Model Serving requires mlflow (not mlflow-skinny) in conda.yaml and cannot build the container image otherwise.”Source: docs.databricks.com
Checkpoint 4 of 9· Check yourself
Every request to a custom pyfunc endpoint is slow. Its predict method calls torch.load on a large weights file before scoring. Which change addresses the cause?
load_context runs once when the model loads, and predict runs on every request. Heavy objects such as weights belong in load_context. Otherwise every request pays the loading cost again.
“anything that needs to be loaded just one time for the model to operate should be defined in this function”Source: docs.databricks.com
3.Creating the endpoint and its recorded creator
There are three ways to create an endpoint, and they accept the same configuration:
- the Serving UI (Serving > Create serving endpoint)
- the REST API (
POST /api/2.0/serving-endpoints) - the MLflow Deployments SDK (
get_deploy_client("databricks").create_endpoint(...))
The configuration is a served_entities list. Each entry names a model with its full three-level Unity Catalog name in entity_name and gives the version in entity_version. Endpoint names may not start with databricks-, because that prefix is reserved for Databricks' own endpoints.
{
"name": "uc-model-endpoint",
"config":
{
"served_entities": [
{
"name": "ads-entity",
"entity_name": "catalog.schema.my-ads-model",
"entity_version": "3",
"min_provisioned_concurrency": 4,
"max_provisioned_concurrency": 12,
"scale_to_zero_enabled": false
}
]
}
}A new endpoint starts as Not Ready. Databricks first packages the model and its environment and then provisions compute, which takes about ten minutes and longer for large or dependency-heavy models. When it finishes, the response shows state.ready as READY and the served entity's deployment as DEPLOYMENT_READY. The response also contains a creator field, which has consequences for the rest of the endpoint's life.
Whoever calls create becomes the endpoint's recorded creator, permanently. The endpoint uses that identity to read Unity Catalog resources. Model grants work as described in Unity Catalog privileges on registered models. For serving, the recorded creator needs USE CATALOG on the catalog, USE SCHEMA on the schema and EXECUTE on the model. If the model declares function dependencies, it also needs EXECUTE on those functions. These grants are checked when the endpoint is created and again on every update. If one is missing, the request fails with PERMISSION_DENIED, even when the person submitting the update has every permission.
The creator can't be changed, so a revoked grant or a deactivated account leaves the endpoint unable to accept updates. The fix is to delete the endpoint and recreate it under a service principal that holds the grants and is a current workspace member. To avoid this, create production endpoints as a long-lived service principal owned by the team, not as a personal user.
Checkpoint 5 of 9· Fill the gap
This REST body creates an endpoint that serves version 3 of a model registered in Unity Catalog. What value goes in the blank?
{
"name": "uc-model-endpoint",
"config":
{
"served_entities": [
{
"name": "ads-entity",
"entity_name": " ? ",
"entity_version": "3",
"min_provisioned_concurrency": 4,
"max_provisioned_concurrency": 12,
"scale_to_zero_enabled": false
}
]
}
}Each served entity names a Unity Catalog model by its full three-level name (catalog.schema.model) in entity_name. The version goes separately in entity_version, so it does not belong in a models:/ URI here. A bare or two-level name does not identify the model in Unity Catalog.
Source: docs.databricks.com4.Workload type, concurrency and scale to zero
Each served entity gets compute along two settings. The workload type sets the hardware and memory per unit of concurrency. Workload size or a provisioned-concurrency range sets how many requests the entity can process at the same time. Choose the hardware first.
| workload_type | Hardware | Memory | Choose it when |
|---|---|---|---|
| CPU | Standard CPU | 4GB per concurrency | Typical scikit-learn or XGBoost models |
| CPU_MEDIUM | Same CPU hardware | 8GB per concurrency | The model needs more memory than CPU gives; you accept less concurrency per worker |
| CPU_LARGE | Same CPU hardware | 16GB per concurrency | The model needs even more memory per worker |
| GPU_SMALL | 1xT4 | 16GB per concurrency | The code actually runs inference on the GPU |
| GPU_MEDIUM | 1xA10G | 24GB per concurrency | A larger GPU model |
A GPU workload costs something in three places, and the model has to be able to use the GPU for that cost to pay off. Your code must place inference on the GPU; MLflow does this automatically only for the PyTorch and Transformers flavors. GPU container builds take longer than CPU builds, and GPU endpoints autoscale more slowly than CPU endpoints. A plain regression model that never puts work on a GPU gets the slower builds and autoscaling with nothing in return.
Size concurrency from traffic, not from a guess. Estimate it as queries per second × model execution time in seconds. At 20 QPS and 0.3 s per prediction, that is 6 requests in flight. You can set the size in two ways. workload_size picks a preset range: Small handles 0-4 concurrent requests, Medium 8-16 and Large 16-64. Alternatively, set min_provisioned_concurrency and max_provisioned_concurrency, which must both be multiples of 4. The endpoint autoscales within that range. It scales up almost immediately when traffic rises and scales down every five minutes. On GPU workloads, concurrency also sets the number of replicas, which is concurrency divided by 4, so a minimum of 12 provisions 3 replicas.
Scale to zero (scale_to_zero_enabled) releases all compute after 30 minutes of inactivity. While no traffic arrives, the endpoint costs nothing. The first request afterwards hits a cold start, usually 10-20 seconds and sometimes minutes, and there is no SLA on how long it takes. GPU capacity in particular isn't guaranteed to come back promptly. Use scale to zero for development endpoints and for models queried only occasionally, where a slow first request is acceptable. Leave it off for any production endpoint with steady traffic or a latency target. For steady moderate traffic, size the entity for its actual concurrency, for example Medium, and keep it running.
Checkpoint 6 of 9· Match them up
Match each workload_type to the served model it fits best.
Tap a term, then the definition that fits it.
CPU, CPU_MEDIUM and CPU_LARGE run on the same CPU hardware and give 4, 8 and 16GB per concurrency, so the larger types trade concurrency for memory. A GPU type is worth its slower builds and autoscaling only when the code places inference on the GPU.
“The CPU_MEDIUM and CPU_LARGE workload types let you trade concurrency for more memory per worker on the same CPU hardware.”Source: docs.databricks.com
Checkpoint 7 of 9· Exam question
A served entity hosts a custom regression model behind a production endpoint that receives a steady, moderate volume of prediction requests around the clock, with no long idle periods. Which configuration choice best fits this workload?
Correct answer: A — Set workload_size to Medium and leave scale_to_zero_enabled set to false, since capacity is not guaranteed once an endpoint scales down and this traffic pattern never truly goes idle.
- A. Sizing the served entity at Medium provisions enough capacity for steady moderate traffic, and disabling scale-to-zero avoids the unguaranteed cold-start capacity risk that scale-to-zero introduces for an endpoint that is effectively always busy.
- B. Enabling scale-to-zero is meant for endpoints with genuine idle windows; against continuous traffic it repeatedly tears down and reprovisions compute, adding cold-start latency instead of saving cost.
- C. GPU workload types accelerate deep-learning-style matrix operations, but a scikit-learn regression model does not use the GPU compute path, so this setting adds cost without improving latency.
- D. Pinning minimum and maximum provisioned concurrency to the same fixed value removes autoscaling headroom, so the endpoint cannot absorb bursts above that fixed level without throttling requests.
5.Updating a live endpoint to a new model version
To move an endpoint to a new model version, you change its served entities. You never delete and recreate it. In the UI this is Edit endpoint. Through the API it is PUT /api/2.0/serving-endpoints/{name}/config, and in the MLflow Deployments SDK it is client.update_endpoint_config(...), which takes the same parameters as the REST API. You can change most of the configuration this way, including entity versions, compute and traffic, but not the endpoint name.
"served_entities": [
{
"entity_name": "catalog.schema.my-ads-model",
"entity_version": "5",
"workload_size": "Small",
"scale_to_zero_enabled": true
}
],
"traffic_config":
{
"routes": [
{
"served_model_name": "my-ads-model-5",
"traffic_percentage": 100
}
]
}Updates don't cause downtime. Version 4 keeps serving every request until version 5 is ready, and only then does traffic move to the new version. You pay for that safety: until the switch completes, both configurations are billed. If the update fails, for example because a package won't install or the recorded creator has lost EXECUTE, the old configuration stays active as if nothing had happened. A failed update can therefore look like a successful one. Check the endpoint's state after every update. Only one update can run at a time, and you can cancel an update in progress from the endpoint's page in the Serving UI.
The same update mechanism supports a challenger. List both versions as served entities on one endpoint, and add a traffic_config whose routes give each entity a traffic_percentage, for example 90 for the current version and 10 for the challenger. Live traffic is then split so the two can be compared on the same requests, while production continues to run on the incumbent. Two separate endpoints with the client always calling one of them never tests the challenger on live traffic. Replacing the incumbent outright takes away the safe default. Routing options and limits are covered in splitting traffic between served entities. After deployment, the application sends requests as described in deploying and querying a model for real-time inference.
Checkpoint 8 of 9· Fill the gap
An endpoint is already live. This MLflow Deployments SDK code changes its served entity and traffic routes in place, without deleting or recreating the endpoint. Which method goes in the blank?
endpoint = client. ? (
endpoint=f"{endpointname}",
config={
"served_entities": [
{
"entity_name": f"{catalog}.{schema}.{model_name}",
"entity_version": "1",
"workload_size": "Small",
"scale_to_zero_enabled": True
}
],
"traffic_config": {
"routes": [
{
"served_model_name": f"{model_name}-1",
"traffic_percentage": 100
}
]
}
}update_endpoint_config is the SDK equivalent of PUT /api/2.0/serving-endpoints/{name}/config. It edits the served entities and traffic of the existing endpoint, and the old configuration keeps serving until the new one is ready. create_endpoint would set up a separate endpoint instead of updating this one.
Source: docs.databricks.comCheckpoint 9 of 9· Exam question
An ML engineer wants to compare a new challenger model version against the current production model on the same endpoint, sending most live traffic to the current version while a small share reaches the challenger for evaluation. Which endpoint configuration achieves this?
Correct answer: A — Define both model versions as served entities on one endpoint, then add a traffic_config with routes that send most traffic to the current entity and the rest to the challenger entity.
- A. Listing both versions as served entities and adding a traffic_config with per-route traffic percentages is exactly how Model Serving performs canary and A/B routing, letting most requests hit the proven version while a small share reaches the challenger.
- B. Two independent endpoints do not share traffic-splitting logic, so the challenger never receives any of the live traffic being compared, which defeats the purpose of an A/B test on the same endpoint.
- C. Serving only the challenger sends it all production traffic immediately with no comparison against the current version, which is the opposite of a controlled traffic split.
- D. Without a traffic_config block, Model Serving does not default to an even split between served entities, so traffic distribution between the two versions would be undefined rather than intentional.
Practise it for real
Take a custom pyfunc model from logging to a live endpoint, then roll the endpoint to a second version without downtime.
1.Log a PythonModel with mlflow.pyfunc.log_model, passing pip_requirements that pin the full mlflow package and registered_model_name set to a catalog.schema.model_name in Unity Catalog.
Why: Databricks Runtime ML would otherwise record mlflow-skinny, and the serving container would fail to build.
You should see: Version 1 of the model appears in Unity Catalog with a signature.
2.Run mlflow.models.predict on the model URI with an input example and env_manager="virtualenv".
Why: It rebuilds the logged dependencies in an isolated environment that simulates serving, so dependency errors show up before you deploy.
You should see: Predictions are printed to stdout with no install or import errors.
3.Call get_deploy_client("databricks").create_endpoint with one served entity for version 1, workload_size Small, and scale_to_zero_enabled True for this test endpoint.
Why: This creates the endpoint and records you, or the service principal you run as, as its permanent creator.
You should see: The endpoint is listed as Not Ready, then READY after about ten minutes.
4.Log and register version 2, then call update_endpoint_config with a served entity for version 2 and a traffic_config route sending 100 percent to it.
Why: Updating the served entities rolls the endpoint to the new version while version 1 keeps serving.
You should see: The endpoint's configuration update state stays in progress until version 2 is ready, then the endpoint returns to NOT_UPDATING with version 2 serving.
Stuck? Get a nudge
If the update fails with PERMISSION_DENIED, check the endpoint's recorded creator, not your own grants.
Sources
(6)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Model Serving can deploy any Python model or custom code as a production-grade API using CPU or GPU compute resources.”
↩︎ What a Model Serving endpoint gives a custom model“application-level dependencies must be explicitly specified in your MLflow model.”
↩︎ Packaging custom logic as a pyfunc model“Signatures are necessary for logging models to the Unity Catalog.”
↩︎ Packaging custom logic as a pyfunc model“This process can take approximately 10 minutes, but may take longer depending on model complexity, size, and dependencies.”
↩︎ Creating the endpoint and its recorded creator“The CPU_MEDIUM and CPU_LARGE workload types let you trade concurrency for more memory per worker on the same CPU hardware.”
↩︎ Workload type, concurrency and scale to zero“you must make sure that your code is set up so that predictions are run on the GPU”
↩︎ Workload type, concurrency and scale to zero“Autoscaling for GPU serving takes longer than for CPU serving.”
↩︎ Workload type, concurrency and scale to zero“provisioned concurrency = queries per second (QPS) * model execution time (s)”
↩︎ Workload type, concurrency and scale to zero“Endpoints scale up almost immediately with increased traffic and scale down every five minutes to match reduced traffic.”
↩︎ Workload type, concurrency and scale to zero“allows them to scale down to zero after 30 minutes of inactivity.”
↩︎ Workload type, concurrency and scale to zero“If model computation takes longer than 597 seconds, requests will time out.”
↩︎ Workload type, concurrency and scale to zero“Databricks performs a zero-downtime update of endpoints by keeping the existing endpoint configuration up until the new one becomes ready.”
↩︎ Updating a live endpoint to a new model version“you are billed for both the old and new endpoint configurations until the transition is complete.”
↩︎ Updating a live endpoint to a new model version“Scaling up from zero usually takes 10-20 seconds, but can sometimes take minutes. There is no SLA on scale from zero latency.”
↩︎ Exam trap 1“After the model is logged, register it in the Unity Catalog (recommended) or the workspace registry.”
↩︎ Checkpoint - 2.
“Model Serving provides low-latency, high-uptime managed REST endpoints with serverless autoscaling.”
↩︎ What a Model Serving endpoint gives a custom model - 3.https://docs.databricks.com/aws/en/machine-learning/model-serving/deploy-custom-python-codeOfficial docs
“load_context - anything that needs to be loaded just one time for the model to operate should be defined in this function.”
↩︎ Packaging custom logic as a pyfunc model“predict - this function houses all the logic that is run every time an input request is made.”
↩︎ Packaging custom logic as a pyfunc model“Model Serving requires mlflow (not mlflow-skinny) in conda.yaml and cannot build the container image otherwise.”
↩︎ Packaging custom logic as a pyfunc model“Model Serving requires mlflow (not mlflow-skinny) in conda.yaml and cannot build the container image otherwise.”
↩︎ Exam trap 3“anything that needs to be loaded just one time for the model to operate should be defined in this function”
↩︎ Checkpoint - 4.https://docs.databricks.com/aws/en/machine-learning/model-serving/model-serving-pre-deployment-validationOfficial docs
“MLflow rebuilds your model's logged dependencies in an isolated environment that simulates the serving runtime”
↩︎ Packaging custom logic as a pyfunc model - 5.https://docs.databricks.com/aws/en/machine-learning/model-serving/create-manage-serving-endpointsOfficial docs
“Endpoint names cannot use the databricks- prefix.”
↩︎ Creating the endpoint and its recorded creator“USE CATALOG on the catalog, USE SCHEMA on the schema, EXECUTE on the model”
↩︎ Creating the endpoint and its recorded creator“you must delete the endpoint and recreate it under a service principal that has the required permissions and is a current workspace member.”
↩︎ Creating the endpoint and its recorded creator“Use a long-lived service principal owned by your team as the endpoint creator.”
↩︎ Creating the endpoint and its recorded creator“Available sizes are Small for 0-4 requests, Medium 8-16 requests, and Large for 16-64 requests.”
↩︎ Workload type, concurrency and scale to zero“Concurrency values must be multiples of 4.”
↩︎ Workload type, concurrency and scale to zero“The number of replicas equals the concurrency value divided by 4.”
↩︎ Workload type, concurrency and scale to zero“Scale to zero is not recommended for production endpoints, as capacity is not guaranteed when scaled to zero.”
↩︎ Workload type, concurrency and scale to zero“While there is an update in progress, another update cannot be made.”
↩︎ Updating a live endpoint to a new model version“Updates to the endpoint configuration can fail.”
↩︎ Updating a live endpoint to a new model version“You can serve multiple models or model versions from a single endpoint and control the traffic split between them.”
↩︎ Key concept“Databricks records the calling identity as the endpoint's creator.”
↩︎ Exam trap 2“Until the new configuration is ready, the old configuration keeps serving prediction traffic.”
↩︎ Exam trap 4“Updates fail with PERMISSION_DENIED if the recorded creator is no longer a workspace member, even when the caller has valid permissions.”
↩︎ Prediction - 6.https://docs.databricks.com/aws/en/machine-learning/model-serving/serve-multiple-models-to-serving-endpointOfficial docs
“split traffic between different models to compare their performance and facilitate A/B testing”
↩︎ Updating a live endpoint to a new model version