What you will be able to do
- Choose between plain-text and secrets-based environment variables for a served model
- Create a secret scope and secret, then reference it in an endpoint configuration with the {{secrets/scope/key}} syntax
- Attach an AWS instance profile to a serving endpoint and explain its limitations
- Pick the right credential mechanism for a given external resource
1.Plain-text vs secrets-based environment variables
Models often have to call things outside Databricks, such as an OpenAI endpoint, another SaaS API or an external storage location. Model Serving passes configuration to the model through environment variables, set per served entity in the environment_vars field. You can set them in the Serving UI (under Advanced configurations), the REST API, the WorkspaceClient SDK or the MLflow Deployments SDK, when you create or update an endpoint. Each served model can have up to 50 environment variables.
There are two kinds. Plain-text variables are for values that don't need to be hidden. Secrets-based variables are for credentials: you store the API key or token as a Databricks secret, and Model Serving fetches it when the endpoint serves requests. Databricks recommends this pattern for deploying the OpenAI and LangChain MLflow model flavors to serving.
A secret reference has to use the form {{secrets/scope/key}}. When the endpoint serves requests, Databricks looks up the secret by scope and key and assigns it to the environment variable name you chose, which the model code then reads. One permission requirement applies: the endpoint creator must have READ access to every secret the configuration references.
{
"name": "endpoint-name",
"config": {
"served_entities": [
{
"entity_name": "model-name",
"entity_version": "1",
"workload_size": "Small",
"scale_to_zero_enabled": "true",
"environment_vars": {
"OPENAI_API_KEY": "{{secrets/my_secret_scope/my_secret_key}}"
}
}
]
}
}Checkpoint 1 of 6· Fill the gap
This PUT /api/2.0/serving-endpoints/{name}/config body adds a secret to an existing endpoint. Which field name completes it?
{
"served_entities": [
{
"entity_name": "model-name",
"entity_version": "2",
"workload_size": "Small",
"scale_to_zero_enabled": "true",
" ? ": {
"OPENAI_API_KEY": "{{secrets/my_secret_scope/my_secret_key}}"
}
}
]
}Plain-text and secrets-based variables both go in the same environment_vars field. The {{secrets/...}} syntax in the value is what makes one a secret.
Source: docs.databricks.comCheckpoint 2 of 6· Exam question
An engineer needs a Model Serving endpoint to call an external, non-Databricks API using an API key, and that key must never appear in plaintext in the endpoint configuration. What should the engineer do?
Correct answer: A — Store the API key as a Databricks secret, then reference it in an endpoint environment variable using `{{secrets/scope/key}}`; the creator needs READ access to that secret.
- A. Secrets-based environment variables are the correct mechanism for sensitive credentials: the key is stored in a Databricks secret scope, referenced with the `{{secrets/scope/key}}` syntax, and the endpoint creator must hold READ access to that secret for the reference to resolve.
- B. Plain-text environment variables are intended for non-sensitive configuration and are not encrypted the way secrets-based variables are, so putting a raw API key there exposes it in the endpoint configuration.
- C. Automatic authentication passthrough only manages credentials for Databricks-managed resources declared through MLflow's resource classes; it has no mechanism for issuing credentials to third-party, non-Databricks APIs.
- D. Hardcoding a credential into the model artifact's source code exposes it to anyone with read access to the logged model and provides no way to rotate or revoke it without redeploying the model.
2.Preparing the secret scope
The {{secrets/scope/key}} reference only works once the scope and secret exist. A secret scope is a named collection of secrets, stored in an encrypted database that Databricks owns and manages. Databricks recommends aligning scopes to roles or applications rather than individuals, so a scope per serving application fits well. The user who creates a scope gets MANAGE on it by default. You then use ACLs to grant other principals access. A service principal is identified by its applicationId value, a user by their email address and a group by its group name.
The workflow runs in this order: create the scope with databricks secrets create-scope my_secret_scope, add the secret with databricks secrets put-secret my_secret_scope my_secret_key, grant access with databricks secrets put-acl <scope-name> <principal> <permission> where needed, and then pass the reference in the endpoint configuration, either at creation or as an update.
The grant step matters when someone other than the scope's creator creates the endpoint. Because the endpoint creator needs READ access to each referenced secret, a service principal that creates the endpoint needs its own ACL on the scope, for example databricks secrets put-acl my_secret_scope <applicationId> READ. A put request for a principal that already has a permission overwrites that permission level. To check the result, databricks secrets get-acl <scope-name> <principal> fails if no ACL exists for that principal.
Databricks also lets you store and govern secrets as Unity Catalog securable objects with a three-level namespace. This lesson covers the secret-scope workflow that the {{secrets/scope/key}} reference uses.
Checkpoint 3 of 6· Put it in order
Put these steps for wiring an API key into a serving endpoint in order
- 1.Reference {{secrets/scope/key}} in the served entity's environment_vars
- 2.Put the API key into the scope under a key name
- 3.Create a secret scope
The scope has to exist before a secret can be put into it, and the endpoint configuration can only reference a secret that already exists.
“The secret information and the name of the environment variable can then be passed to your endpoint configuration”Source: docs.databricks.com
Model answer: the service principal, as the endpoint creator, needs READ access to the referenced secret. You grant it with databricks secrets put-acl <scope-name> <principal> <permission>, using READ as the permission and the service principal's applicationId value as the principal.
Sources3
3.Instance profiles for AWS resources
Environment variables carry API keys and tokens. For AWS resources, an endpoint can instead carry an instance profile. Attaching one lets the model access any AWS resource that the instance profile's policy allows. Because model serving endpoints run on serverless compute, the instance profile's IAM role must have a trust relationship configured for serverless compute. If you are reusing a profile set up for serverless SQL, check that its access policy gives your models the access they need. You add the profile in Advanced configurations in the Serving UI, or with the instance_profile_arn field on the served entity. The endpoint creator's permission on the profile is checked when the endpoint is created.
{
"served_entities": [{
"entity_name": "ads1",
"entity_version": "2",
"workload_size": "Small",
"scale_to_zero_enabled": true,
"instance_profile_arn": "arn:aws:iam::<aws-account-id>:instance-profile/<instance-profile-name-2>"
}]
}Limitations to remember:
- Network: data access is authenticated with STS temporary security credentials, and those credentials can't get around network restrictions. - Edited role: if someone edits the profile's IAM role in Databricks Settings, running endpoints keep using the old role until the endpoint is updated. - Deleted profile: if someone deletes the profile from Settings, running endpoints that use it are not affected. - Feature lookups: automatic feature lookups use the same instance profile you added to the endpoint.
Worked scenario: you attach an instance profile that was originally set up for a serverless SQL warehouse, and endpoint creation or use fails with "IAM role does not have the required trust relationship". The cause is the role's trust relationship, because model serving endpoints run on serverless compute. The fix is to set up the trust relationship for serverless compute as described for serverless SQL warehouses, and to confirm the access policy gives your models the access they need.
Checkpoint 4 of 6· Check yourself
An admin narrows the IAM role behind an instance profile in Databricks Settings. A serving endpoint that uses this profile is already running. What happens?
Running endpoints keep the IAM role they started with. The change takes effect only when the endpoint is updated.
“endpoints running with the instance profile continue to use the old IAM role until the endpoint updates.”Source: docs.databricks.com
Sources4
4.Choosing the right mechanism
Each mechanism matches a different kind of resource. Agents add one more. When a deployed agent gets authentication errors reaching resources such as AI Search indexes or LLM endpoints, first check that it was logged with the resources needed for automatic authentication passthrough. To fix missing resources, log the agent again and redeploy. If you authenticate manually through environment variables instead, those manual settings override any automatic authentication configuration.
| Need | Mechanism | Configured via |
|---|---|---|
| Non-sensitive setting, e.g. ENABLE_FEATURE_TRACING | Plain-text environment variable | environment_vars |
| API key for OpenAI or another external model | Secrets-based environment variable | environment_vars with {{secrets/scope/key}} |
| AWS resources allowed by an IAM role | Instance profile | instance_profile_arn |
| Unity Catalog model and its functions | Recorded creator's grants | USE CATALOG, USE SCHEMA, EXECUTE |
Checkpoint 5 of 6· Match them up
Match each requirement to the mechanism that meets it
Tap a term, then the definition that fits it.
Credentials go in secrets, AWS access goes through instance profiles, non-sensitive flags go in plain text, and agent authentication passthrough depends on the resources logged with the agent.
“You must store credentials like your API key or other tokens as a Databricks secret.”Source: docs.databricks.com
Checkpoint 6 of 6· Exam question
A team is deploying an agent whose Unity Catalog table queries must respect each individual end user's own row- and column-level permissions rather than a single shared identity's permissions. Which configuration achieves this on a Model Serving endpoint?
Correct answer: A — Enable on-behalf-of-user authentication, build the client with `ModelServingUserCredentials` inside `predict`, and declare `api_scopes` in an `AuthPolicy` at logging time.
- A. On-behalf-of-user authentication is the feature designed for this scenario: initializing `ModelServingUserCredentials` inside the prediction function (since user identity is only known at request time) and declaring the needed API scopes lets each request enforce that specific user's own Unity Catalog permissions.
- B. Automatic authentication passthrough with a declared resource grants the same service-principal-level access to every caller of the endpoint, so it cannot differentiate row- or column-level permissions between individual end users.
- C. Referencing a single service principal's token means every request authenticates as that one identity, which enforces uniform access rather than each end user's individual row- and column-level entitlements.
- D. Granting a broad metastore admin role bypasses access checks entirely rather than applying each user's own permissions, which is the opposite of enforcing per-user row- and column-level governance.
Sources5
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Any environment variable whose value points at a secret path is resolved as a secret.Why is that wrong?
Only the {{secrets/scope/key}} syntax is resolved. Any other value is passed to the model as plain text.
Covered in Plain-text vs secrets-based environment variables
2.Only the users who query the endpoint need access to the secret scope.Why is that wrong?
The endpoint creator is the one who must have READ access to the referenced secrets.
Covered in Plain-text vs secrets-based environment variables
3.An instance profile's credentials can reach AWS resources even when a network restriction blocks the path.Why is that wrong?
The STS temporary credentials only authenticate data access. They can't bypass network restrictions.
Covered in Instance profiles for AWS resources
Practise it for real
Give a served model an external API key through a Databricks secret, without the key ever appearing in the endpoint configuration.
1.Run
databricks secrets create-scope my_secret_scope.Why: Secrets live inside a named scope, and the endpoint reference points to a scope and key.
You should see:
databricks secrets list-scopesshows my_secret_scope.2.Run
databricks secrets put-secret my_secret_scope my_secret_keyand enter the API key when prompted.Why: This stores the credential encrypted in Databricks instead of in code or config.
You should see:
databricks secrets list-secrets my_secret_scopelists my_secret_key.3.If a service principal will create the endpoint, grant it access with
databricks secrets put-acl my_secret_scope <applicationId> READ.Why: The endpoint creator needs READ access to every secret the configuration references.
You should see:
databricks secrets get-acl my_secret_scope <applicationId>returns the ACL.4.Create or update the endpoint with
"environment_vars": {"OPENAI_API_KEY": "{{secrets/my_secret_scope/my_secret_key}}"}on the served entity.Why: The {{secrets/...}} syntax tells Model Serving to fetch the value when it serves requests.
You should see: The endpoint deploys, and the model reads OPENAI_API_KEY at runtime.
Stuck? Get a nudge
If the model receives the literal string instead of the key, check the curly braces. Without them the value is treated as plain text.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/machine-learning/model-serving/store-env-variable-model-servingOfficial docs
“Model Serving supports plain text environment variables and secrets-based environment variables using Databricks secrets.”
↩︎ Plain-text vs secrets-based environment variables“Databricks recommends this feature for deploying OpenAI and LangChain MLflow model flavors to serving.”
↩︎ Plain-text vs secrets-based environment variables“During model serving, the secrets are retrieved from Databricks secrets by the secret scope and key.”
↩︎ Plain-text vs secrets-based environment variables“The secrets based environment variable must be provided using the following syntax: {{secrets/scope/key}}.”
↩︎ Exam trap 1“The endpoint creator must have READ access to the Databricks secrets being referenced in the configs.”
↩︎ Exam trap 2“Otherwise, the environment variable is considered a plain text environment variable.”
↩︎ Prediction“The secret information and the name of the environment variable can then be passed to your endpoint configuration”
↩︎ Checkpoint“You must store credentials like your API key or other tokens as a Databricks secret.”
↩︎ Checkpoint - 2.
“Environment variables | Per served model | 50.”
↩︎ Plain-text vs secrets-based environment variables - 3.https://docs.databricks.com/aws/en/security/secretsOfficial docs
“Databricks recommends aligning secret scopes to roles or applications rather than individuals.”
↩︎ Preparing the secret scope“A user is specified using their email address, a service principal using its applicationId value, and a group using its group name.”
↩︎ Preparing the secret scope“By default, scopes are created with MANAGE permission for the user who created the scope.”
↩︎ Preparing the secret scope“Making a put request for a principal that already has an applied permission overwrites the existing permission level.”
↩︎ Preparing the secret scope“To store and govern secrets as Unity Catalog securable objects using the three-level namespace”
↩︎ Preparing the secret scope - 4.https://docs.databricks.com/aws/en/machine-learning/model-serving/add-model-serving-instance-profileOfficial docs
“Your instance profile's IAM role must have a trust relationship configured for serverless compute.”
↩︎ Instance profiles for AWS resources“The endpoint creator's permission to an instance profile is validated at endpoint creation time.”
↩︎ Instance profiles for AWS resources“Look up features using the same instance profile that you added to the serving endpoint.”
↩︎ Instance profiles for AWS resources“IAM role does not have the required trust relationship”
↩︎ Instance profiles for AWS resources“STS temporary security credentials are used to authenticate data access. It can't bypass any network restriction.”
↩︎ Exam trap 3“endpoints running with the instance profile continue to use the old IAM role until the endpoint updates.”
↩︎ Checkpoint - 5.
“verify that it was logged with the necessary resources for automatic authentication passthrough”
↩︎ Choosing the right mechanism“Manual settings override any automatic authentication configurations.”
↩︎ Choosing the right mechanism