CertSafari
    Snowflake SnowPro Advanced: MLOps Engineer (MLA-B01)· Lessons

    Domain 2 · Lesson 8/17

    Snowflake ML experiment tracking, HPO and custom models

    Track experiments and metadata.

    21 min read
    8% of exam
    5 sources
    Published 20 Oct 2026
    Docs as of 4 Oct 2026

    What you will be able to do

    • Create an experiment, open runs with start_run, and explain when a run becomes immutable and which privileges this requires
    • Choose correctly between log_param, log_metric with a step, log_artifact and log_model when recording training metadata
    • Compare, query, retrieve and clean up runs, and recognise the per-schema, per-experiment and per-run limits
    • Configure a Tuner with a search algorithm, sampling functions and TunerConfig, and read its TunerResults
    • Wrap an external model in a CustomModel subclass using ModelContext and @inference_api methods
    • Tie a registered model version to the exact Dataset version it was trained on and trace that lineage

    Key concept

    Experiment run — A run is one training attempt inside a named experiment. It holds the parameters, step-indexed metrics, artifacts and linked model versions from that attempt, and it is frozen once completed. Comparing, tuning and reproducing models in Snowflake ML all come down to comparing runs.

    1.Experiments, runs and the privileges behind them

    Snowflake ML Experiments give model development a structure: a schema contains experiments, an experiment contains runs, and each run holds its own parameters, metrics and artifacts. One experiment usually covers one modelling question, such as which gradient-boosting setup best predicts churn. Each attempt at answering it is a run. Snowsight builds its run lists, charts and comparisons from this metadata. Snowflake doesn't restrict what you store as a run artifact, so anything that helps you evaluate a model can go in: a confusion-matrix image, a feature-importance file or a log.

    Experiments need snowflake-ml-python 1.19.0 or later. They also need a database and schema that already exist, because that is where run information is stored. You can create an experiment in Snowsight under AI & ML » Experiments » New Experiment by choosing a name and the database and schema for its run artifacts. In code, an ExperimentTracking(session) instance with set_experiment("my_experiment") selects the experiment that later calls write to.

    The privilege model follows from where the data is stored. The role needs CREATE EXPERIMENT on the schema that will hold the run artifacts, plus USAGE (or another privilege) on the parent database and schema so it can reach that schema. Privileges for other objects don't count. CREATE DATASET covers Datasets, and CREATE STAGE covers stages. Neither one lets a role create an experiment.

    A run starts with start_run(name) on the tracking instance. It returns a Run that works as a context manager. Snowflake recommends the with form because leaving the block completes the run automatically, which makes the run's scope easy to see in the code. If you start a run without with, close it yourself with end_run("my_run"). Completion matters because a completed run becomes immutable and shows as finished in Snowsight. You can't add metrics to it afterwards, so finish all logging before the with block ends.

    The recommended run scope: the run completes when the with block exitspython
    with exp.start_run("my_run"):
      # .. Train your model and log artifacts

    Checkpoint 1 of 9· Fill the gap

    You want a run that is completed automatically, and so becomes immutable, as soon as the block exits. Which method on the ExperimentTracking instance completes this sample?

    with exp. ? ("my_run"):
      # .. Train your model and log artifacts

    Checkpoint 2 of 9· Exam question

    Which privileges does a role need to create an experiment and write runs into a given schema using Experiment Tracking?

    Sources1

    2.Logging parameters, step metrics, artifacts and models

    Run metadata comes in three kinds, and each answers a different question. Parameters are constant inputs to training, such as a learning rate, an optimizer name or a batch size. They are set once and don't change during the run. Metrics are measured at a model step, and each value is recorded against that step. This is what lets Snowsight draw a curve: if each epoch is a step, log_metric("val_loss", value, step=epoch) inside the training loop adds one point per epoch to the same metric. If you leave out step, the value is logged at step 0, and one call after training gives you a single point, not a history.

    Manual logging inside an active run: parameters have no step, metrics are recorded at a steppython
    # Log model parameters with the log_param(...) or log_params(...) methods
    exp.log_param("learning_rate", 0.01)
    exp.log_params({"optimizer": "adam", "batch_size": 64})
    
    # Log model metrics with the log_metric(...) or log_metrics(...) methods
    exp.log_metric("loss", 0.3, step=100)
    exp.log_metrics({"loss": 0.4, "accuracy": 0.8}, step=200)
    Which logging call records what
    CallWhat it recordsWhere it shows up
    log_param / log_paramsConstant training inputs, one value per nameOptional parameter columns in the runs list; string columns from list_params
    log_metric / log_metrics (step=…)Values measured at a model step; default step is 0One column per metric in the runs list, metric charts, float columns from list_metrics
    log_artifactA local file uploaded under an artifact_pathRun artifacts, retrievable with LIST/GET on snow://experiment/…
    log_modelA model logged to the registry from the active runLinked model versions in the run and comparison views
    set_live_logging_status(True)stdout and stderr of the notebook or SPCS workloadThe default event table, shown in the run's Logs tab

    The other channels have specific jobs. log_artifact('/tmp/file.txt', artifact_path='artifacts') uploads a file, but Snowsight doesn't read numbers out of it to chart them. Charts come from logged metrics only. exp.log_model(model, model_name="my_model", ...) logs the model to the registry from inside the run. You pass either signatures or sample_input_data, and the run then shows the resulting model version as linked. Live logging sends stdout and stderr from a Snowflake Notebook or another SPCS workload, such as an ML Job, to the default event table, and you read them in the run's Logs tab. It captures console output, not metric history, and it doesn't work in legacy notebooks.

    For XGBoost, LightGBM and Keras you can skip manual calls and use autologging. You build a callback such as SnowflakeLightgbmCallback(exp, model_name="name", model_signature=sig), pass it to fit inside start_run, and the framework's parameter and metric updates are logged to the active run. Manual logging is still needed for models that don't support autologging and for pre-trained models.

    Checkpoint 3 of 9· Check yourself

    A data scientist trains a Keras-free custom model for 30 epochs inside an active run and wants Snowsight to chart validation loss as a curve with one point per epoch. Which logging approach gives that chart?

    Checkpoint 4 of 9· Exam question

    A team trains an XGBoost model in a Container Runtime notebook and wants every epoch's validation loss stored so that the loss curve can be plotted per run in Snowsight. Which call pattern achieves this?

    Sources1

    3.Comparing runs, retrieving artifacts and staying under the limits

    Experiments are evaluated in Snowsight under AI & ML » Experiments. The runs list shows the run name, status, created date and one column per metric, and you can add parameters as extra columns. You can select up to five runs and press Compare. The comparison view puts their metadata, parameters, metrics and model version information side by side. Linked model versions come from the Model Lineage feature, which requires Enterprise Edition or above.

    In code, list_metrics() and list_params() return Snowpark DataFrames with one row per run. Both include a run_name column, and each logged metric becomes a float column while each parameter becomes a string column. Because they are DataFrames, you can join metrics to parameters on run_name, filter for runs with loss under a threshold, or call to_pandas() and analyse them locally. Pass run_name="RUN_1" to get a single run's values. Artifacts are stored at a snow://experiment/ path and can be retrieved with SQL at any time, during or after a run.

    Listing a run's logs artifacts and downloading one file to /tmpsql
    LIST snow://experiment/my_experiment/versions/my_run/logs;
    GET snow://experiment/my_experiment/versions/my_run/logs/log0.txt file:///tmp;

    Plan cleanup from the start, because each limit applies at a different level. A nightly retraining job that adds one run to the same experiment reaches 500 runs in about sixteen months. After that, new runs fail until old ones are removed. ALTER EXPERIMENT my_experiment DROP RUN my_run; removes a single run. DROP EXPERIMENT my_experiment; removes the experiment and all of its run artifacts. The metric and parameter limits apply to each run, not to the schema or the experiment.

    Experiment limits and the scope each applies to
    LimitScopeCleanup when reached
    500 experimentsPer schemaDROP EXPERIMENT (removes all its run artifacts)
    500 runsPer experimentALTER EXPERIMENT … DROP RUN
    1000 unique parametersPer runLog fewer distinct parameter names
    200 unique metricsPer runLog fewer distinct metric names

    Experiments have no separate charge, but they still cost something. You pay storage for run artifacts and warehouse credits for the Snowsight charts, because virtual warehouses render them. Dropping runs you no longer need keeps you under the run limit and also cuts storage.

    Checkpoint 5 of 9· Check yourself

    For about sixteen months, a nightly retraining job has added one run to the same experiment. Each run logs about 40 metrics. Now new runs fail to start. What is the most likely cause, and what is the fix?

    Checkpoint 6 of 9· Exam question

    A team runs nightly retraining and each night adds a new run to the same experiment. After about a year, start_run begins failing. Which documented limit is the most likely cause?

    Sources1

    4.Hyperparameter optimization with the Tuner API

    The Snowflake ML HPO API runs many training trials in parallel and keeps the best configuration. It works with any model: open-source frameworks and Snowflake ML's own distributed training APIs both fit. You launch it from a Snowflake Notebook on Container Runtime. By default the trials spread across the CPU or GPU cores of a single node in the SPCS compute pool, and they can scale to multiple nodes. Choosing those resources belongs to compute pools and Container Runtime. Data reaches the trials through a dataset_map, a dictionary that maps names such as "train" and "test" to DataConnector objects.

    A tuning job has three parts: a search algorithm, a search space and a configuration. GridSearch tries every combination in a grid you define. RandomSearch samples the space at random and works well when the space is large or mixes continuous and discrete values. BayesOpt uses a probabilistic model of earlier trials to choose the next configuration. The search space uses sampling functions to describe each hyperparameter's range and type.

    Search-space sampling functions in snowflake.ml.modeling.tune
    FunctionSamplesTypical use
    uniform(lower, upper)A continuous value, uniformlyDropout rates, regularization strengths
    loguniform(lower, upper)A value uniformly in log spaceLearning rates spanning orders of magnitude
    randint(lower, upper)An integer, lower inclusive and upper exclusiveNumber of layers, epochs or estimators
    choice(options)One value from a listCategorical options such as activation functions

    TunerConfig sets the goal and the budget. metric names the value to optimize, such as accuracy. mode is "max" or "min". search_alg picks the strategy, num_trials sets how many configurations to try, and max_concurrent_trials sets how many run at once. Bayesian search needs earlier results to choose the next trial, so max_concurrent_trials=1 gives each trial the most information, at the cost of running them one after another.

    The training function connects your code to the tuner. Inside it, get_tuner_context() returns the hyperparameters for the current trial and the dataset map. The function then trains and calls report with the metrics and the model. If a trial doesn't report its metrics, the search algorithm has nothing to learn from.

    A training function reads its trial's hyperparameters and reports metrics back through TunerContextpython
    def train_func():
      tuner_context = get_tuner_context()
      config = tuner_context.get_hyper_params()
      dm = tuner_context.get_dataset_map()
      ...
      tuner_context.report(metrics={"accuracy": accuracy}, model=model)

    tune.Tuner(train_func, search_space, tuner_config).run(dataset_map=dataset_map) distributes the trials and returns a TunerResults object. Its results attribute is a pandas DataFrame with the metrics and configuration of every trial. best_result is the row for the best trial, and best_model is that trial's model instance, if one was reported. Because data comes in through Snowflake Datasets or DataFrames, ML lineage is captured automatically. In practice, the best trial's settings and metrics are what you log as parameters and metrics in a run, so the tuned model can be compared with the rest of the experiment.

    Checkpoint 7 of 9· Fill the gap

    In a BayesOpt tuning job, every trial has to return its accuracy and trained model to the tuner so the search algorithm can choose the next configuration. Which TunerContext method fills the blank?

    def train_func():
      tuner_context = get_tuner_context()
      config = tuner_context.get_hyper_params()
      dm = tuner_context.get_dataset_map()
      ...
      tuner_context. ? (metrics={"accuracy": accuracy}, model=model)

    Sources2

    5.Bringing external models in with the CustomModel API

    The Model Registry logs built-in model types directly. For anything else, such as a PyCaret pipeline, a pickled model from an open-source repository or a hand-written algorithm, you subclass snowflake.ml.model.custom_model.CustomModel. The subclass gets a ModelContext holding everything inference needs and exposes methods decorated with @custom_model.inference_api. Each decorated method takes a pandas DataFrame and returns one.

    A CustomModel that reads its wrapped model from the context inside an inference_api methodpython
    # Initialize ModelContext with an in-memory model object
    # my_model can be any supported model type (e.g., sklearn, xgboost, lightgbm, and others)
    model_context = custom_model.ModelContext(
        my_model=my_model,
    )
    
    # Define a custom model class that utilizes the context
    class ExampleBringYourOwnModel(custom_model.CustomModel):
        def __init__(self, context: custom_model.ModelContext) -> None:
            super().__init__(context)
    
        @custom_model.inference_api
        def predict(self, input: pd.DataFrame) -> pd.DataFrame:
            # Use the model with key 'my_model' from the context to make predictions
            model_output = self.context['my_model'].predict(input)
            return pd.DataFrame({'output': model_output})

    ModelContext takes keyword arguments of two kinds. An in-memory object of a supported type, such as an XGBoost or scikit-learn model, can be passed directly, and Snowflake ML serializes it for you. A string file path points to a pickle, a JSON configuration or a parameter file. Your __init__ opens it from self.context['my_file_path']. Both kinds are packaged with the model. Helper code your model imports, such as a local scoring_utils/ package, is included through code_paths when you log the model.

    Always reach the wrapped model through the context. If predict uses a notebook variable directly, or sets self.model = model instead of self.context['my_model'], a second copy of the model is captured in a closure. The serialized model files can then grow much larger.

    Write every inference method to handle many rows. Inference methods can also accept options that control behavior, such as temperature or max_tokens. Each option must be keyword-only (declared after *), must have a type annotation (int, float, str, bool, bytes, datetime.datetime, or a list of a supported type) and must have a default value. When the model is logged, these options are added to its signature automatically. Callers can override them at inference time or leave them out and get the defaults.

    Checkpoint 8 of 9· Fill the gap

    This wrapper should expose predict as a method the registered model can serve. Which custom_model decorator fills the blank?

    # Initialize ModelContext with an in-memory model object
    # my_model can be any supported model type (e.g., sklearn, xgboost, lightgbm, and others)
    model_context = custom_model.ModelContext(
        my_model=my_model,
    )
    
    # Define a custom model class that utilizes the context
    class ExampleBringYourOwnModel(custom_model.CustomModel):
        def __init__(self, context: custom_model.ModelContext) -> None:
            super().__init__(context)
    
        @custom_model. ? 
        def predict(self, input: pd.DataFrame) -> pd.DataFrame:
            # Use the model with key 'my_model' from the context to make predictions
            model_output = self.context['my_model'].predict(input)
            return pd.DataFrame({'output': model_output})

    Sources3

    6.Tying a model version to its Dataset version

    A run is only reproducible if you know exactly which rows trained the model. A source table that changes every day can't tell you that. Snowflake Datasets (versioned, immutable snapshots) can, because each version is a fixed point-in-time snapshot. Training from a specific Dataset version and logging the model so the registry records that link means the model version always points to data that still exists unchanged. Six months later you can retrain on exactly the same rows.

    Lineage for a model is recorded when the model is logged to the Model Registry. The simplest way to get it is to call ds.read.to_snowpark_dataframe() on the selected version. That DataFrame points to the version's materialized data, not to the original query. Train with Snowpark ML on it, then log the model. Training on a Snowpark DataFrame records lineage automatically, which completes the graph from source data through feature views and the Dataset to the model.

    Lineage is not automatic when you train on a pandas DataFrame, train without Snowpark ML, or train outside Snowflake. In those cases, pass a Snowpark DataFrame backed by the Dataset (or other source object) as sample_input_data to log_model, and the registry records the link. Two caveats apply. Only objects created after ML Lineage is enabled in the account carry lineage, and lineage information is not replicated.

    To read lineage back, call the lineage method on a ModelVersion, Dataset or FeatureView. direction is "upstream" or "downstream" (downstream is the default), and domain_filter limits results to types such as "dataset" or "model". model_version.lineage(direction="upstream") shows where a model's training data came from. The call below runs the other way, from a Dataset down to the models trained on it. Exploring lineage from Python requires the VIEW LINEAGE account privilege, which ACCOUNTADMIN has by default and can grant to other roles. Each artifact's Snowsight page also has a Lineage tab.

    Finding every model trained on data from a given Datasetpython
    my_dataset.lineage(direction="downstream", domain_filter=["model"])

    Checkpoint 9 of 9· Check yourself

    Your team pulls a Dataset version into pandas with to_pandas(), trains a scikit-learn model on it and logs the model to the Model Registry. Working as ACCOUNTADMIN, you call model_version.lineage(direction="upstream"), and no Dataset comes back. What should the team change so the model version links to the Dataset version it was trained on?

    Sources45

    Sources

    (5)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “requires the CREATE EXPERIMENT privilege on the schema where run artifacts are stored, plus USAGE or any other privilege on the parent database”
      ↩︎ Experiments, runs and the privileges behind them
      “Completing a run makes it immutable and presents it as finished in Snowsight.”
      ↩︎ Experiments, runs and the privileges behind them
      “Snowflake Experiments require snowflake-ml-python version 1.19.0 or later.”
      ↩︎ Experiments, runs and the privileges behind them
      “You can choose to represent a training epoch as a corresponding step.”
      ↩︎ Logging parameters, step metrics, artifacts and models
      “When live logging is enabled, stdout and stderr output from the notebook is written to the Snowflake default event table.”
      ↩︎ Logging parameters, step metrics, artifacts and models
      “You can autolog training information for XGBoost, LightGBM, or Keras models during model training.”
      ↩︎ Logging parameters, step metrics, artifacts and models
      “You can select up to five runs in your experiment.”
      ↩︎ Comparing runs, retrieving artifacts and staying under the limits
      “list_metrics includes a run_name column and one float column per logged metric”
      ↩︎ Comparing runs, retrieving artifacts and staying under the limits
      “The charts in the UI are powered by virtual warehouses.”
      ↩︎ Comparing runs, retrieving artifacts and staying under the limits
      “Viewing linked model versions is part of the Model Lineage feature, which is only available for customers on Enterprise Edition and above.”
      ↩︎ Comparing runs, retrieving artifacts and staying under the limits
      “Each experiment consists of a series of runs, which are metadata and artifacts from your training.”
      ↩︎ Key concept
      “Parameters are constant inputs to the training model, while metrics are evaluated at a model step.”
      ↩︎ Exam trap 1
      “Runs are limited to 1000 unique parameters and 200 unique metrics.”
      ↩︎ Exam trap 2
      “metrics are evaluated at a model step. You can choose to represent a training epoch as a corresponding step.”
      ↩︎ Checkpoint
    2. 2.
      “The workload scales out to the CPU or GPU cores that are available on a single node in the SPCS compute pool.”
      ↩︎ Hyperparameter optimization with the Tuner API
      “The Snowflake ML HPO API requires the training metrics and hyperparameters from each training run to optimize the hyperparameters effectively.”
      ↩︎ Hyperparameter optimization with the Tuner API
      “results: A Pandas DataFrame containing metrics and configurations for every trial.”
      ↩︎ Hyperparameter optimization with the Tuner API
      “efficient data ingestion via Snowflake Datasets or Dataframes and automatic ML lineage capture”
      ↩︎ Hyperparameter optimization with the Tuner API
      “The HPO API evaluates every possible combination of hyperparameters.”
      ↩︎ Prediction
    3. 3.
      “Serializable models trained using external tools or obtained from open source repositories can be used with CustomModel.”
      ↩︎ Bringing external models in with the CustomModel API
      “Files can be serialized models, configuration files, or files containing parameters.”
      ↩︎ Bringing external models in with the CustomModel API
      “When this model is logged, the parameters are automatically included in the model signature.”
      ↩︎ Bringing external models in with the CustomModel API
      “Include additional code with your model using code_paths.”
      ↩︎ Bringing external models in with the CustomModel API
      “Accessing the model directly captures a second copy of the model in a closure, which results in significantly larger model files during serialization.”
      ↩︎ Exam trap 3
      “Don’t assume that the input DataFrame will always contain a single row.”
      ↩︎ Prediction
    4. 4.
      “Each version is an immutable, point-in-time snapshot of the data managed by the Dataset.”
      ↩︎ Tying a model version to its Dataset version
      “The trained model can then be logged to Snowflake Model Registry, automatically completing the ML Lineage graph linking source data, feature views, datasets”
      ↩︎ Tying a model version to its Dataset version
    5. 5.
      “Lineage for models is captured when the model is logged to the Model Registry.”
      ↩︎ Tying a model version to its Dataset version
      “Training a model using Snowpark ML automatically generates lineage records if the model is trained from a Snowpark DataFrame.”
      ↩︎ Tying a model version to its Dataset version
      “Users need the VIEW LINEAGE privilege to explore lineage from Python APIs.”
      ↩︎ Tying a model version to its Dataset version
      “Only objects created after the ML Lineage feature is enabled in your account contain lineage information.”
      ↩︎ Tying a model version to its Dataset version
      “passing a Snowpark DataFrame backed by the source data object as sample_data to the Model Registry’s log_model method”
      ↩︎ Exam trap 4
      “passing a Snowpark DataFrame backed by the source data object as sample_data to the Model Registry’s log_model method”
      ↩︎ Checkpoint

    Ready to test yourself?

    Practise the 29 questions on this subdomain.

    Spotted a mistake, or was something unclear? Tell us.