What you will be able to do
- Write FeatureLookup objects that name a feature table, its features and the lookup keys that join it to label data
- Build a training set with create_training_set using label, exclude_columns, output names and default values, then train on load_df()
- Log the trained model with fe.log_model so it keeps its feature lookups and lineage for inference
- Explain why manual joins, edited training DataFrames and plain MLflow logging break automatic feature lookup
Key concept
Feature-aware model — A model trained from a training set built from feature tables and logged with the feature client's log_model. It stores which tables, features and keys it was trained on. At inference time it can fetch those features itself when given only the primary keys.
1.What training from a feature table gives you
You don't train on a feature table by reading it and joining it yourself. You declare which features you want and how they join to your labels. Feature Engineering then builds the training DataFrame for you. The declaration matters more than the DataFrame. Once the model is trained on it and logged through the feature client, the model retains references to those features. Every later step depends on those references: batch scoring, Model Serving and lineage.
Here is the payoff. When the model is used for inference, the caller only needs to supply the primary keys, for example customer_id. The model works out the rest from what it stored at training time. It knows which tables to read, which columns to take and how to join them. You never have to rebuild the join by hand, so training and inference cannot drift apart.
The same references feed governance. Catalog Explorer automatically tracks and displays the tables and functions used to create a model. This is the model→feature-table link described in Lineage and discovery in Unity Catalog. The link exists only because the training set and the logging call carried the feature metadata. If you train on a DataFrame you assembled yourself, there is nothing to record.
One hard limit applies to every feature-aware model: it can use at most 50 tables and 100 functions for training.
Checkpoint 1 of 8· Check yourself
A team declares FeatureLookups and builds a training set with create_training_set. They train on it and log the model through the feature client's log_model. When the model is later used for scoring, what does the caller need to supply?
Logging with the feature client stores references to the tables, features and keys used in training. So at inference the model fetches and joins the features itself, and the caller only passes the primary keys.
“The caller only needs to provide the primary key of the features used in the model”Source: docs.databricks.com
2.Naming features with FeatureLookup
A FeatureLookup names one feature table, the features you want from it, and the key used to join it. Many models share a feature table, and few need every column, so you make one lookup per table, or more than one when the same table is joined on different keys. The example below takes two purchase features from a customer table and one category feature from a product table.
feature_lookups = [
FeatureLookup(
table_name='ml.recommender_system.customer_features',
feature_names=['total_purchases_30d', 'total_purchases_7d'],
lookup_key='customer_id'
),
FeatureLookup(
table_name='ml.recommender_system.product_features',
feature_names=['category'],
lookup_key='product_id'
)
]| Parameter | What it does |
|---|---|
| table_name | The feature table to read, by its three-level Unity Catalog name (for example ml.recommender_system.customer_features) |
| feature_names | One feature name, a list of names, or None to take all features except the primary keys as they exist when the training set is created |
| lookup_key | Column(s) in your label DataFrame that join to the table's primary keys, in the same type and order |
| output_name | Replaces the feature's column name in load_df(); needed when the same feature is looked up twice |
| rename_outputs | A mapping from feature names to new output column names |
| default_values | Values to fill in when the table has no computed value for an ID; keys must use the renamed names if rename_outputs is set |
| timestamp_lookup_key | The label-side time column used for a point-in-time join against a time series feature table |
lookup_key is a list of columns in your DataFrame, not in the feature table. The names can differ. A label column called cid can join to a primary key called customer_id. What must line up is the type and the order: the lookup columns must match the table's primary keys in the order those keys were declared when the table was created. The join is positional, so listing two keys in the wrong order joins the wrong columns to each other.
Time columns follow different rules. Timestamp columns are not used as a lookup_key. For a time series feature table, you pass the label-side time column as timestamp_lookup_key. The system then performs an as-of join and takes the latest feature value at or before each label's timestamp, so values from the future cannot leak into training. How that column becomes part of the primary key is covered in Time series feature tables (timeseries column in the primary key). There is also a failure you may hit here: if a table has a DATE or TIMESTAMP primary key that was never declared as a timeseries column, create_training_set() returns an error. The fix is either to declare the column as timeseries or, if you only need exact matching, to change its type to STRING.
Two more parameters handle common problems. The first is name collisions. Suppose customer and product tables both have a height feature. Give each lookup a different output_name, such as customer_height and product_height. Do the same when one feature is joined twice, for example temperature looked up once by pickup_zip and once by dropoff_zip. The second problem is missing values. default_values fills in a value, such as age 18 or membership_tier "bronze", when the table has nothing for an ID.
FeatureLookup(
table_name = 'main.default.table',
feature_names = ['materialized_feature_value'],
lookup_key = 'id',
rename_outputs={"materialized_feature_value": "feature_value"},
default_values={
"feature_value": 0
}
)A FeatureLookup is the right tool when the features already exist in a table. A FeatureFunction solves a different problem. It is for features that depend on information available only at inference time, which it combines with stored feature values. Recomputing columns that a table already holds wastes that table. Reading the whole table with read_table and calling .join() yourself also gives you the right rows, but nothing records which features the model used.
Checkpoint 2 of 8· Fill the gap
This FeatureLookup renames a feature with rename_outputs. Which key belongs in default_values so that missing IDs get the value 0?
FeatureLookup(
table_name = 'main.default.table',
feature_names = ['materialized_feature_value'],
lookup_key = 'id',
rename_outputs={"materialized_feature_value": "feature_value"},
default_values={
" ? ": 0
}
)Once rename_outputs is set, default_values has to use the renamed output name (feature_value), not the column name in the table (materialized_feature_value). The lookup key (id) is never a default_values key.
Source: docs.databricks.comCheckpoint 3 of 8· Check yourself
A feature table was created with its primary keys declared as (store_id, product_id). Your label DataFrame calls these columns sid and pid. Which lookup_key joins the table correctly?
lookup_key lists columns from your own DataFrame, and their names can differ from the table's keys. The join is positional, though, so the columns must match the primary keys in type and in the order the keys were declared.
“The type and order of lookup_key columns in the DataFrame must match the type and order of the primary keys”Source: docs.databricks.com
3.Building the training set with create_training_set
create_training_set combines the lookups with a DataFrame of raw training rows. That DataFrame must contain every lookup key used by the lookups and the label column. The label argument names the ground-truth column. exclude_columns drops columns you need for the join but don't want as model inputs, usually the ID columns. Calling load_df() on the resulting TrainingSet returns the joined DataFrame.
training_set = fe.create_training_set(
df=training_df,
feature_lookups=feature_lookups,
label='rating',
exclude_columns=['customer_id', 'product_id']
)
training_df = training_set.load_df()The join is a left join from your training DataFrame to each feature table. Every label row is kept. The DataFrame returned by load_df() has one column for each looked-up feature, plus all of your original columns except the excluded ones. A label row whose key has no match in the feature table stays in the result with an empty feature value, unless you set a default for that feature. For unsupervised work such as clustering customers by interests, pass label=None.
For that reason, you train on the DataFrame returned by load_df() as it is. Splitting off the label to form X_train and y_train is fine, because the label is not an input at inference. Any feature engineering you want the model to keep belongs upstream, in the feature table itself, where training and scoring both read it. If it lives only in your notebook between load_df() and fit(), it is lost at inference.
Checkpoint 4 of 8· Fill the gap
customer_id and product_id are needed to join the feature tables, but they should not be model inputs. Which argument keeps them out of the DataFrame that load_df() returns?
training_set = fe.create_training_set(
df=training_df,
feature_lookups=feature_lookups,
label='rating',
? =['customer_id', 'product_id']
)
training_df = training_set.load_df()exclude_columns removes columns that the join needs but the model should not use, usually the ID columns. load_df() keeps all other original columns and adds one column per looked-up feature. lookup_key and feature_names belong to FeatureLookup, not to create_training_set.
Source: docs.databricks.comCheckpoint 5 of 8· Exam question
A team maintains a Unity Catalog feature table customer_features with primary key customer_id, containing columns avg_purchase_value and days_since_last_order. They have a separate DataFrame labels_df with customer_id and a churned label, and want to train a churn model using only the two named features joined by primary key while preserving feature lineage for the resulting model. Which approach best meets this requirement?
Correct answer: A — Instantiate a FeatureEngineeringClient, call create_training_set on labels_df with a FeatureLookup for customer_features naming customer_id as the lookup key and the two feature columns, then call .load_df() to materialize the joined training DataFrame.
- A. Using
create_training_setwith aFeatureLookupautomatically joins the named features by key and, critically, embeds lineage metadata so the resulting model tracks which feature table and columns it depends on. - B. Manually joining
read_tableoutput like this returns the correct feature values but bypassesFeatureLookup, so the resulting model has no recorded lineage back to the feature table for later governance. - C. A
FeatureFunctionis for deriving new features computed at request time from input arguments, not for pulling existing stored columns, so it doesn't fit a scenario that just needs two already-stored features by key. - D. Round-tripping the feature table through a CSV export defeats the purpose of a governed feature store and loses the primary-key-based lookup and lineage tracking that Feature Engineering in Unity Catalog provides.
Sources1
4.Packaging feature metadata with fe.log_model
Training produces an ordinary model object. Logging decides whether it stays tied to its features. fe.log_model takes the same TrainingSet object you built earlier as its training_set argument. It stores the feature lookups next to the model: the tables, the features and the keys. That stored metadata is what lets the model fetch its own features later.
training_df = training_set.load_df().toPandas()
# "training_df" columns ['total_purchases_30d', 'category', 'rating']
X_train = training_df.drop(['rating'], axis=1)
y_train = training_df.rating
model = linear_model.LinearRegression().fit(X_train, y_train)
fe.log_model(
model=model,
artifact_path="recommendation_model",
flavor=mlflow.sklearn,
training_set=training_set,
registered_model_name="recommendation_model"
)flavor must be an MLflow Python flavor that matches the model type. The supported list includes scikit-learn, PyTorch, SparkML, LightGBM, XGBoost and TensorFlow Keras (through mlflow.keras), and custom pyfunc models are also accepted. registered_model_name registers the model in the same call.
If you log with plain mlflow.sklearn.log_model instead, the model is saved correctly but without its feature lookups. It no longer knows which tables it came from, so inference callers must send every feature value themselves. Unity Catalog also has no model→feature-table link to display. Writing the table name into the model description or saving the joined DataFrame as an artifact with mlflow.log_artifact does not fix this. Neither gives the model lookups it can run, and a saved DataFrame only holds the feature values from training time. The client matters as well. For tables in Unity Catalog, you build the training set and log the model with the same FeatureEngineeringClient, as explained in FeatureEngineeringClient vs legacy FeatureStoreClient. The lookups are packaged by that client's log_model, and no other call adds them.
The stored lookups are what make the inference paths work. Model Serving uses the entity IDs in each request to look up precomputed features in the online store, so a client sends only customer_id. For batch scoring, fe.score_batch(model_uri=..., df=...) reads the lookups stored with the model and joins current feature values by key before predicting. The input DataFrame needs only the key columns. Batch scoring is covered fully in its own lesson, Score a model using features from a feature store table.
The input columns, then the looked-up features, then the prediction: customer_id, product_id, total_purchases_30d, category, prediction. You never ran a join yourself. score_batch did it from the lookups stored with the model.
Checkpoint 6 of 8· Fill the gap
The team wants Model Serving and fe.score_batch to fetch this model's features when callers send only customer_id and product_id. Which object's log_model has to be called here?
training_df = training_set.load_df().toPandas()
# "training_df" columns ['total_purchases_30d', 'category', 'rating']
X_train = training_df.drop(['rating'], axis=1)
y_train = training_df.rating
model = linear_model.LinearRegression().fit(X_train, y_train)
? .log_model(
model=model,
artifact_path="recommendation_model",
flavor=mlflow.sklearn,
training_set=training_set,
registered_model_name="recommendation_model"
)Only the feature client's log_model (fe, the FeatureEngineeringClient) packages the TrainingSet's lookups with the model, and that lets the model fetch features from keys alone. Plain MLflow logging saves the model without any feature metadata, so callers would have to send every feature value. It also leaves no model→feature-table lineage.
Source: docs.databricks.comCheckpoint 7 of 8· Exam question
After training a scikit-learn model on a training set created via fe.create_training_set(...), a data scientist wants Model Serving to automatically look up the same features from the feature table at inference time, so client requests only need to supply customer_id rather than every feature value. Which step accomplishes this?
Correct answer: A — Pass the original TrainingSet object returned by create_training_set to fe.log_model(model=model, artifact_path='model', flavor=mlflow.sklearn, training_set=training_set, registered_model_name=...), so the logged model packages the feature lookup metadata needed for automatic retrieval.
- A. Passing the
TrainingSetintofe.log_modelpackages theFeatureLookupdefinitions with the model, so a compatible serving environment can fetch the current feature values by key automatically rather than requiring the caller to supply them. - B. Plain
mlflow.sklearn.log_modelhas no awareness of feature lookups, so the resulting model still expects every feature value in the request payload; documentation elsewhere does not change what the endpoint accepts. - C. Storing the training DataFrame as a static artifact captures only a snapshot of features at training time and provides no mechanism for a serving endpoint to perform fresh, key-based lookups on each request.
- D. Registering directly from the artifact URI still yields a model expecting full feature vectors as input, and manually wiring
fe.read_tablecalls into the endpoint is not how Feature Engineering automatic lookup is configured.
Checkpoint 8 of 8· Exam question
A model was trained and logged using fe.log_model with its TrainingSet. A data scientist now needs to run batch inference on a new Spark DataFrame inference_df that only contains the primary key column customer_id plus any raw label columns, without manually re-joining the feature table. Which call correctly produces predictions with features re-attached automatically?
Correct answer: A — fe.score_batch(model_uri='models:/churn_model/3', df=inference_df), which reads the feature lookup metadata stored with the model to join the current feature values by customer_id before scoring.
- A.
score_batchreads theFeatureLookupmetadata embedded in the model byfe.log_modeland performs the join against the current feature table automatically, which is exactly what is needed here. - B. A plain pyfunc model has no built-in awareness of feature tables or lookup keys, so calling
predicton a DataFrame missing feature columns raises a schema mismatch rather than triggering an automatic join. - C. Manually reading the feature table and joining it works mechanically, but it discards the point of
fe.log_modelandscore_batch, which is to avoid hand-writing the same join logic at every inference call site. - D. Selecting the full feature table and filtering it without an explicit join key still requires manual alignment with
inference_df, and passing it straight into a plain sklearn model without a join reintroduces exactly the manual work the feature store workflow avoids.
Practise it for real
Train a scikit-learn model from two Unity Catalog feature tables and log it so it keeps its feature lookups.
1.Create a FeatureEngineeringClient and define two FeatureLookups: total_purchases_30d from ml.recommender_system.customer_features keyed on customer_id, and category from ml.recommender_system.product_features keyed on product_id.
Why: Each lookup names a table, its features and the label-side join key.
You should see: A feature_lookups list with two FeatureLookup objects and no errors.
2.Inside mlflow.start_run(), call fe.create_training_set with your label DataFrame (columns customer_id, product_id, rating), label='rating' and exclude_columns=['customer_id', 'product_id'].
Why: The label DataFrame must contain every lookup key and the label. Excluding the IDs keeps them out of the model inputs.
You should see: A TrainingSet object.
3.Call training_set.load_df().toPandas(), split off rating as y_train, and fit LinearRegression on the remaining columns without any further changes.
Why: Changes made after load_df() would not be reapplied at inference.
You should see: X_train has the columns total_purchases_30d and category.
4.Call fe.log_model with model, artifact_path, flavor=mlflow.sklearn, training_set=training_set and a registered_model_name.
Why: This stores the lookups with the model and writes the model→feature-table lineage.
You should see: A registered model version whose lineage in Catalog Explorer shows both feature tables.
Stuck? Get a nudge
If create_training_set errors on a DATE or TIMESTAMP primary key, check that the column was declared as a timeseries column.
Sources
(3)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/machine-learning/feature-store/train-models-with-feature-storeOfficial docs
“A model can use at most 50 tables and 100 functions for training.”
↩︎ What training from a feature table gives you“or None to look up all features (excluding primary keys) in the feature table”
↩︎ Naming features with FeatureLookup“must match the type and order of the primary keys (excluding timestamp keys) of the reference feature table.”
↩︎ Naming features with FeatureLookup“Timestamp columns should not be used as a lookup_key.”
↩︎ Naming features with FeatureLookup“calls to create_feature_spec(), create_training_set(), and publish_table() return an error.”
↩︎ Naming features with FeatureLookup“Use a unique output_name for each FeatureLookup output.”
↩︎ Naming features with FeatureLookup“you can specify default values for features to handle cases where the Feature Store does not have a computed feature value for an ID.”
↩︎ Naming features with FeatureLookup“The training DataFrame must contain all lookup keys from the set of feature lookups”
↩︎ Building the training set with create_training_set“it creates a training dataset by performing a left join”
↩︎ Building the training set with create_training_set“It preserves all columns of the DataFrame provided to create_training_set except those excluded using exclude_columns.”
↩︎ Building the training set with create_training_set“Set label=None when creating a TrainingSet for unsupervised learning models.”
↩︎ Building the training set with create_training_set“The model type must have a corresponding python_flavor in MLflow.”
↩︎ Packaging feature metadata with fe.log_model“call automatically retrieves the required features from Feature Store before scoring the model.”
↩︎ Packaging feature metadata with fe.log_model“When you then train the model, it retains references to those features.”
↩︎ Key concept“the modifications are not applied when you use the model for inference. This decreases the performance of the model.”
↩︎ Exam trap 1“You must log the model using the log_model method of FeatureEngineeringClient”
↩︎ Exam trap 2“If the feature columns are renamed using the rename_outputs parameter, default_values must use the renamed feature names.”
↩︎ Exam trap 3“must match the type and order of the primary keys (excluding timestamp keys) of the reference feature table.”
↩︎ Exam trap 4“The type and order of lookup_key columns in the DataFrame must match the type and order of the primary keys”
↩︎ Checkpoint“the modifications are not applied when you use the model for inference. This decreases the performance of the model.”
↩︎ Prediction - 2.
“The caller only needs to provide the primary key of the features used in the model”
↩︎ What training from a feature table gives you“Tables and functions that were used to create the model are automatically tracked and displayed.”
↩︎ What training from a feature table gives you“The system performs an as-of timestamp join, using the timestamp_lookup_key you specify.”
↩︎ Naming features with FeatureLookup“A feature might depend on information that is only available at the time of inference.”
↩︎ Naming features with FeatureLookup“To package a model with feature metadata, use FeatureEngineeringClient.log_model (for Feature Engineering in Unity Catalog)”
↩︎ Packaging feature metadata with fe.log_model“the model serving endpoint automatically uses the entity IDs in the request data to look up pre-computed features from the online store”
↩︎ Packaging feature metadata with fe.log_model - 3.
“To work with feature tables in Unity Catalog, use FeatureEngineeringClient.”
↩︎ Packaging feature metadata with fe.log_model