Publishing Offline Feature Tables to a Databricks Online Store

1.Preparing an offline table for publishing

An online feature table always comes from an offline one. Once a Databricks online store, created with create_online_store, reaches the AVAILABLE state, "The publish_table API synchronizes data from your offline feature table to online store created using create_online_store API." The offline Delta table has to meet three requirements first, whether or not it has a time series column:

- Primary key constraint: required for publishing to an online store. - Non-nullable primary keys: primary key columns cannot contain NULL values. - Change Data Feed enabled: required for the CONTINUOUS and TRIGGERED publish modes.

The last one matters because the sync is incremental. The publish pipeline reads changes from the offline table instead of copying the whole table every time. The docs give two DDL statements to get an existing table ready:

-- Enable CDF if not already enabled
ALTER TABLE catalog.schema.your_feature_table
SET TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true');

-- Ensure primary key columns are not nullable
ALTER TABLE catalog.schema.your_feature_table
ALTER COLUMN user_id SET NOT NULL;

-- Enable CDF if not already enabled
ALTER TABLE catalog.schema.your_feature_table
SET TBLPROPERTIES (' ? ' = 'true');

Sources1

2.Time series tables: the online copy keeps only the latest value

This is one of the clearest differences between the two kinds of table. Offline, a time series feature table supports point-in-time joins: create_training_set and score_batch do an as-of join so that each training row sees the feature values that were known at its timestamp. Online, a real-time endpoint usually wants the current value. So when the offline table has a time series designation, rows that share a primary key but have different time series values are deduplicated in the publish pipeline.

If an endpoint really needs historical values, create the offline table *without* a time series designation. Then "All rows from the source (offline) table are published without deduplication." Even then the online table does not do as-of queries: "This is an exact-match lookup on a string primary key, not a point-in-time (as-of) or temporal range query." To use a DATE or TIMESTAMP column as a plain lookup key, change the column type to STRING.

Sources12

3.publish_table and the three publish modes

Publishing is a single call on FeatureEngineeringClient. You pass the online store, the offline source_table_name and an online_table_name. The call creates the online table if it doesn't already exist, syncs the feature data from the offline table, and sets up the infrastructure that keeps the online store in sync with the offline table from then on.

from databricks.ml_features.entities.online_store import DatabricksOnlineStore

# Get the online store instance
# For Lakebase Autoscaling projects creating using the Lakebase API or UI,
# `name` is the last part of the resouce name: projects/{online_store_name}
online_store = fe.get_online_store(name="my-online-store")

# Publish the feature table to the online store
fe.publish_table(
    online_store=online_store,
    source_table_name="catalog_name.schema_name.feature_table_name",
    # for online_table_name, the catalog name, schema name, and table name each are limited to a maximum of 63 bytes
    online_table_name="catalog_name.schema_name.online_feature_table_name",
    # `publish_mode` argument is optional and defaults to "TRIGGERED" mode if not specified
)

How fresh the online copy is depends on publish_mode. This parameter replaced the older streaming parameter, and passing streaming=True is equivalent to publish_mode="CONTINUOUS".

Sources1

4.Querying and deleting online tables

Once the published table shows as AVAILABLE, you can work with it in two ways, neither of which looks like the offline Delta table. In the Unity Catalog UI you can open the online table to see sample data and check its schema. In the SQL editor you can run PostgreSQL queries against online feature tables, because the store is Lakebase. To serve the features to applications you create a Feature Serving endpoint. Models trained on Databricks features already carry lineage to those features, and when they're deployed they use Unity Catalog to find the matching online tables.

Deleting is where people most often treat an online table like an ordinary table. The only recommended way is the Databricks SDK, which removes the table from both Unity Catalog and the database. A SQL DROP TABLE, or the SDK command that deletes a synced table, leaves the data in the underlying database storage. Before you delete, make sure no model serving or feature serving endpoint still uses the online features, because removing a published table can break downstream dependencies.

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
w.feature_store. ? (online_table_name="catalog_name.schema_name.online_feature_table_name")

Sources1

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Publishing a time series feature table online keeps its full history, so real-time endpoints get the same point-in-time lookups as training.Why is that wrong?

   With a time series designation, the publish pipeline deduplicates rows and keeps only the latest values per entity. Without one, every row is published, but lookups are exact matches and never as-of queries.

   Covered in Time series tables: the online copy keeps only the latest value

2. 2.Running DROP TABLE on an online feature table removes it completely.Why is that wrong?

   DROP TABLE leaves the data in the underlying database storage. Only the SDK's delete_online_table removes the table from both Unity Catalog and the database.

   Covered in Querying and deleting online tables

3. 3.A Delta feature table with a primary key can be published in the default mode with no further changes.Why is that wrong?

   The default TRIGGERED mode, like CONTINUOUS, requires Change Data Feed on the offline table, and primary key columns must also be non-nullable.

   Covered in Preparing an offline table for publishing