Writing Data to Feature Tables with write_table

1.What you are writing to, and with which client

Databricks Feature Store gives you two ways to author features. With Feature Views, Databricks manages the pipelines for you. With feature tables, you write the values yourself. This lesson is about the second way, and that means you own the code that fills the table and keeps it current. In a workspace enabled for Unity Catalog, a feature table is not a special storage format. It is a Delta table with a primary key, named with the usual three levels: <catalog-name>.<schema-name>.<table-name>. Every example below writes to ml.recommender_system.customer_features.

Before you write anything, check which client you have. The current client is FeatureEngineeringClient, from the databricks-feature-engineering package. It comes pre-installed in Databricks Runtime 13.3 LTS ML and above. On a non-ML runtime you install it yourself with %pip install databricks-feature-engineering. The older FeatureStoreClient is for the legacy Workspace Feature Store only, and the databricks-feature-store package was deprecated as of version 0.17.0.

One more limit: the client runs only on Databricks. You can install it locally to mock its calls in unit tests, but it cannot call Feature Engineering APIs from your laptop.

Sources123

2.The first write: populating a new feature table

The Python workflow has three steps. First, write functions that compute the features. Each one should return a Spark DataFrame with a unique primary key, which can be one column or several. Second, create the table with create_table. Third, populate it with write_table. The DataFrame comes first because create_table can take its schema straight from it.

from databricks.feature_engineering import FeatureEngineeringClient

fe = FeatureEngineeringClient()

# Prepare feature DataFrame
def compute_customer_features(data):
  ''' Feature computation code returns a DataFrame with 'customer_id' as primary key'''
  pass

customer_features_df = compute_customer_features(df)

# Create feature table with `customer_id` as the primary key.
# Take schema from DataFrame output by compute_customer_features
customer_feature_table = fe.create_table(
  name='ml.recommender_system.customer_features',
  primary_keys='customer_id',
  schema=customer_features_df.schema,
  description='Customer features'
)

That call creates an empty table. You can skip the separate write by passing the DataFrame to create_table as df=customer_features_df. The docs say that this "automatically saves the features to the underlying Delta table." For a composite key, pass a list such as primary_keys=['customer_id', 'date'].

You don't have to use Python. If you create the table in Databricks SQL with a PRIMARY KEY constraint, you can then write to it like any other Delta table. A materialized view or streaming table published by Lakeflow pipelines with a primary key is also a feature table, and you write to it like any other pipeline dataset. An existing Delta table becomes writable as a feature table once it has a primary key. Only the table owner can add that constraint, using ALTER TABLE ... ADD CONSTRAINT ... PRIMARY KEY.

Sources2

3.Updating with mode='merge'

After the first load, most writes are updates. The Unity Catalog docs describe two kinds: adding new features, and changing specific rows based on the primary key. Both go through write_table with mode='merge'.

Merge works key by key. Rows whose keys are in the DataFrame get updated. Keys the table doesn't have yet come in as new rows. Rows whose keys aren't in the DataFrame are left alone. That's why a merge write is safe to run on a schedule with only the latest batch of entities.

from databricks.feature_engineering import FeatureEngineeringClient
fe = FeatureEngineeringClient()

customer_features_df = compute_customer_features(data)

fe.write_table(
  df=customer_features_df,
  name='ml.recommender_system.customer_features',
  mode='merge'
)

Merge is also how you add new feature columns. One way is to extend the existing computation function and write its output. That updates the table schema and merges the new values by primary key. The other way is a separate function that returns only the new features. Its DataFrame must still contain the table's primary key, plus the partition keys if any are defined, so each value lands on the right row. Some metadata should not change through writes: the primary key, the partition key, and the name or data type of an existing feature. Changing any of these breaks the downstream training and serving pipelines that read the table.

  mock_write_table.assert_called_once_with(
    name='ml.recommender_system.customer_features',
    df=customer_features_df,
    mode=' ? '
  )

Sources24

4.Keeping features fresh: scheduled jobs and streaming writes

A feature table is only useful while its values are current. Databricks recommends a job that runs your update notebook on a schedule, for example daily. If you already have an unscheduled job, you can convert it to a scheduled one. The notebook just runs the merge write from the previous section.

Batch isn't the only option, because write_table also accepts a streaming DataFrame. In that case it starts a streaming feature pipeline and returns a StreamingQuery object instead of finishing as a one-off write. The call looks the same and still uses merge mode.

customer_transactions = spark.readStream.table("prod.events.customer_transactions")
stream_df = compute_additional_customer_features(customer_transactions)

fe.write_table(
  df=stream_df,
  name='ml.recommender_system.customer_features',
  mode='merge'
)

Each write is also recorded. Feature table metadata tracks the data sources a table was built from, and the notebooks and jobs that created it or wrote to it. That record shows where a feature value came from.

Sources25

5.Writing to time series feature tables

A time series feature table includes a timestamp column in its primary key and marks it with TIMESERIES in SQL, or with timeseries_columns in create_table. You still write to it with write_table and merge mode. There is one rule that ordinary feature tables don't have.

For time series tables, every write must include values for all of the table's features. A DataFrame with just one or two feature columns won't do. The docs say this rule keeps feature values from becoming sparse across timestamps. Streaming writes to time series tables are supported.

fe = FeatureEngineeringClient()
# daily_users_batch_df DataFrame contains the following columns:
# - user_id
# - ts
# - purchases_30d
# - is_free_trial_active
fe.write_table(
  "ml.ads_team.user_features",
  daily_users_batch_df,
  mode="merge"
)

Sources6

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.write_table with mode='merge' makes the table match the DataFrame, so keys missing from the DataFrame get deleted.Why is that wrong?

   Merge updates and inserts rows by primary key. Rows whose keys aren't in the DataFrame are left alone.

   Covered in Updating with mode='merge'

2. 2.FeatureStoreClient.write_table is the way to write to feature tables in Unity Catalog.Why is that wrong?

   Unity Catalog feature tables use FeatureEngineeringClient. FeatureStoreClient is for the legacy Workspace Feature Store.

   Covered in What you are writing to, and with which client

3. 3.Like an ordinary feature table, a time series feature table accepts a merge with only a subset of its feature columns.Why is that wrong?

   Writes to time series feature tables must supply values for every feature, which keeps values from becoming sparse across timestamps.

   Covered in Writing to time series feature tables

4. 4.Since write_table can change the table schema, it's fine to rename a feature or change its data type in a later write.Why is that wrong?

   Feature names, data types, the primary key and the partition key should not change. Changing them breaks downstream training and serving pipelines.

   Covered in Updating with mode='merge'