What you will be able to do
- Explain why Databricks promotes code, not trained models, from one environment to the next
- Trace a change through the dev, staging and production workspaces of the default MLOps Stack
- Name the three components of an MLOps Stack and choose the right
databricks bundle init mlops-stackssetup option for a role - Design a scalable multi-environment setup by using bundle targets, per-environment catalogs and run identities, and define ML assets (experiments, registered models, model serving endpoints) as bundle resources
Key concept
Deploy code, not models — In the MLOps Stacks design, the whole model development process lives as code in a source-controlled repository. What moves from one environment to the next is that code, and each environment uses it to build its own model. The trained model artifact is not copied between environments.
1.Why environments promote code, not models
Most people assume the trained model is what gets promoted. A model is trained in development, so it seems natural to copy it to production. Databricks recommends the opposite. With MLOps Stacks, the entire model development process is implemented, saved and tracked as code in a source-controlled repository. That includes feature engineering, training, testing and deployment notebooks, the pipelines that run them, the workspace used for each stage, and the CI/CD workflows. Because the process is code, it can go through your normal CI/CD process, and deployments become repeatable, predictable and systematic.
The practical payoff shows up at retraining time. If production holds only a copied artifact, retraining means rebuilding the model somewhere else and copying it again. If production holds the code, retraining is just another run of a job that already exists there. This one idea drives the rest of the environment design: dev, staging and prod are places where the same code runs, under different levels of control.
Checkpoint 1 of 4· Check yourself
What does representing the model development process as code let you deploy between environments, instead of the trained model?
Because the process is code, you deploy the code and each environment builds its own model, which also makes retraining easier.
“Representing the model development process as code enables you to deploy the code instead of deploying the model.”Source: docs.databricks.com
Sources1
2.The dev, staging and prod workspace flow
The default MLOps Stack uses three separate Databricks workspaces: development, staging and production. Here is how a change travels through them.
In the development workspace, data scientists iterate on ML code, either on Databricks or on their own machine, and open pull requests (PRs) to update it. Each PR triggers unit tests and integration tests in an isolated staging workspace. When the PR is merged to main, the model training and batch inference jobs that run in staging switch to the latest code straight away. Later, as part of a scheduled release process, the team cuts a release branch and deploys those code changes to production.
People do not deploy to staging and production by hand. A CI/CD runner (GitHub Actions or Azure DevOps) runs notebooks, creates jobs and does other tasks in those workspaces, so all production changes go through automation and only tested code reaches prod. ML engineers set up Databricks service principal secrets for the CI/CD runner. For fully automated workflows, Databricks recommends OAuth machine-to-machine (M2M) authentication rather than the user-to-machine (U2M) login you would use interactively.
Checkpoint 2 of 4· Put it in order
Put the stages of a code change in the default MLOps Stack in order
- 1.A data scientist iterates on ML code in the development workspace and opens a PR
- 2.A release branch is cut and the code changes are deployed to production
- 3.The PR is merged to main, and staging training and batch inference jobs update to the latest code
- 4.The PR triggers unit and integration tests in the isolated staging workspace
Development produces the PR, staging tests it and then runs the merged code, and production receives the code only when a release branch is cut.
“After you merge a PR into main, you can cut a new release branch as part of your scheduled release process”Source: docs.databricks.com
Checkpoint 3 of 4· Check yourself
A data scientist wants to fix a bug in the production training job by editing it directly in the production workspace UI. According to the MLOps Stacks design, how should production changes be made?
The CI/CD component exists so that every production change is automated and tested. Manual edits and manual copies both skip that gate.
“ensuring that all production changes are performed through automation and that only tested code is deployed to prod”Source: docs.databricks.com
No person is present when an automated pipeline runs. Databricks recommends U2M for trying steps out interactively and M2M for fully automated workflows. A service principal also means deployments to staging and production do not depend on any one person's credentials.
3.What an MLOps Stack contains
An MLOps Stack is an MLOps project on Databricks that follows production best practices from the start. Its modular structure lets different ML teams work on the same project independently. The default stack has three components:
- ML code: template notebooks for training, batch inference and similar tasks, so every team uses the same project structure and the code is modular and ready to test. - ML resources as code: resources such as workspaces and pipelines for training and batch inference, defined in Declarative Automation Bundles (formerly called Databricks Asset Bundles, or DABs). Because these definitions are version-controlled, a change like trying a larger instance type for automated retraining is tracked automatically. - CI/CD: GitHub Actions or Azure DevOps workflows that test and deploy both the ML code and the resources.
You create a stack with the Databricks CLI.
databricks bundle init mlops-stacksThe first prompt asks which components to set up, so each role can start with only what it needs. To add the other components later, run the command again. Once created, the project includes role-specific guides. For example, <project-name>/resources/README.md is for data scientists changing production ML resources such as training or inference jobs, and docs/mlops-setup.md is for MLOps or DevOps engineers setting up CI/CD. The table below lists the Databricks tool behind each part of the stack.
| Component | Tool in Databricks |
|---|---|
| ML model development code | Databricks notebooks, MLflow |
| Feature development and management | Feature engineering |
| ML model repository | Models in Unity Catalog |
| ML model serving | Model Serving |
| Infrastructure-as-code | Declarative Automation Bundles |
| Orchestrator | Lakeflow Jobs |
| CI/CD | GitHub Actions, Azure DevOps |
| Data and model performance monitoring | Data profiling |
Checkpoint 4 of 4· Match them up
Match each databricks bundle init mlops-stacks setup option to what it creates
Tap a term, then the definition that fits it.
The first prompt splits the stack by role. Running the command again adds whichever components you skipped.
“CICD_Only - Set up CI/CD components only. This option is for ML engineers to set up infrastructure.”Source: docs.databricks.com
Designing a scalable environment with bundles. The "ML resources as code" component is where environment design happens. A bundle defines named deployment environments called *targets* in databricks.yml, each pointing at its own workspace, catalog and variable values. You write the project once and add a target (for example dev, staging, prod) per environment, so adding an environment means adding a target rather than copying a project. Deploy with databricks bundle deploy -t <target-name>.
A few practices make this scale:
- Isolate environments with Unity Catalog names. Models and tables use the three-level name catalog.schema.object. Pass the catalog in as a bundle variable or from the target (the documented registered model example sets catalog_name: ${bundle.target}), so identical code writes to dev, staging or prod data and models without edits. The catalog reflects the environment and its governance rules, not whether a model version serves traffic; deployment status is managed with model aliases.
- Set the mode per target. mode: development prefixes resource names, tags jobs and pauses schedules and triggers, so work does not affect anyone else. mode: production removes those safety defaults and adds validations.
- Run as a service principal in staging and prod. Set run_as per target so the identity that runs a workflow is decoupled from the one that deployed it. Databricks recommends service principals for production deployments. The exception: run_as is not supported for model serving endpoints.
- Define ML assets as bundle resources. Experiments (experiments), Unity Catalog registered models (registered_models) and serving endpoints (model_serving_endpoints) sit under resources, next to jobs and pipelines, so they are version-controlled and deployed per target. Each declaration mirrors the create request of the matching REST API object, expressed in YAML.
- Validate before deploying. Run databricks bundle validate first; it returns warnings for unknown resource properties.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.In the recommended MLOps workflow, you promote a trained model artifact from the staging registry to the production registry.Why is that wrong?
MLOps Stacks deploys the code that builds the model. Production runs that code to produce its own model, which also makes retraining easy.
Covered in Why environments promote code, not models
2.A pull request's unit and integration tests run in the development workspace where the data scientist wrote the code.Why is that wrong?
In the default stack, PRs trigger tests in a separate, isolated staging workspace. Development is for iterating.
Covered in The dev, staging and prod workspace flow
3.Setting a top-level run_as service principal in a bundle works for every resource, including model serving endpoints.Why is that wrong?
run_as is not supported for model serving endpoints; defining one in a bundle that also configures run_as causes an error.
Covered in What an MLOps Stack contains
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Representing the model development process as code enables you to deploy the code instead of deploying the model.”
↩︎ Why environments promote code, not models“PRs trigger unit tests and integration tests in an isolated staging Databricks workspace.”
↩︎ The dev, staging and prod workspace flow“Resources are defined in Declarative Automation Bundles to facilitate testing, optimization, and version control for the ML environment.”
↩︎ What an MLOps Stack contains“Infrastructure-as-code | Declarative Automation Bundles”
↩︎ What an MLOps Stack contains“Representing the model development process as code enables you to deploy the code instead of deploying the model.”
↩︎ Key concept“enables you to deploy the code instead of deploying the model”
↩︎ Exam trap 1“PRs trigger unit tests and integration tests in an isolated staging Databricks workspace.”
↩︎ Exam trap 2“Deploying the code automates the ability to build the model, making it much easier to retrain the model when necessary.”
↩︎ Prediction“After you merge a PR into main, you can cut a new release branch as part of your scheduled release process”
↩︎ Checkpoint“ensuring that all production changes are performed through automation and that only tested code is deployed to prod”
↩︎ Checkpoint - 2.
“For fully automated workflows, Databricks recommends that you use OAuth machine-to-machine (M2M) authentication instead.”
↩︎ The dev, staging and prod workspace flow“CICD_Only - Set up CI/CD components only. This option is for ML engineers to set up infrastructure.”
↩︎ Checkpoint - 3.
“A bundle defines named deployment environments called targets in databricks.yml, each pointing at its own workspace, catalog, and variable values.”
↩︎ What an MLOps Stack contains - 4.https://docs.databricks.com/aws/en/dev-tools/bundlesOfficial docs
“Definitions and settings for Databricks resources, such as Lakeflow Jobs, Lakeflow pipelines, Dashboards, Model Serving endpoints, MLflow Experiments, and MLflow registered models”
↩︎ What an MLOps Stack contains - 5.
“catalog_name: ${bundle.target}”
↩︎ What an MLOps Stack contains - 6.
“The run_as setting is not supported for model serving endpoints.”
↩︎ What an MLOps Stack contains“The run_as setting is not supported for model serving endpoints.”
↩︎ Exam trap 3 - 7.
“Databricks recommends that you use service principals for production deployments.”
↩︎ What an MLOps Stack contains