What you will be able to do
- Describe how a Spark Connect client encodes DataFrame operations and how results come back
- Explain which Spark APIs a Spark Connect client cannot reach and what the separation buys
- Create a remote session with sc:// through SparkSession.builder.remote, SPARK_REMOTE or --remote
- Predict when an analysis error, a replaced temporary view or a changed UDF variable takes effect under Spark Connect
- State how Databricks Connect relates to Spark Connect and which runtimes it covers
Key concept
Spark Connect — A protocol that splits Spark into a thin client and a remote server. The client only builds unresolved plans and sends them over gRPC. The server analyzes them, executes them and streams the results back, so nothing is resolved until the server sees the plan.
1.A decoupled client for a remote Spark server
Spark Connect separates the code that describes a query from the process that runs it. Apache Spark 3.4 introduced it as a decoupled client-server architecture: your application holds a small client library, and a remote Spark server holds the driver. The client can be embedded almost anywhere, including application servers, IDEs, notebooks and other programming languages, because it no longer has to host Spark itself.
The term usually comes up on Databricks, because many Databricks environments already run on Spark Connect. Python notebooks on standard compute with Databricks Runtime 14.3 and above use it, as do Scala notebooks on 13.3 and above, serverless compute, Databricks Connect and Lakeflow pipelines with environment versions configured. Code that ran fine on a classic cluster can behave differently once it lands on one of these, so the differences in this lesson matter even if you never start a Connect server yourself.
Classic Spark and Spark Connect agree on one thing. Both run transformations lazily, and an action such as show() or collect() triggers the computation. They differ in when a query is analyzed: when column and table names are resolved and checked. Most of what is surprising about Spark Connect follows from that one difference.
Checkpoint 1 of 10· Check yourself
A team moves a PySpark job from a classic cluster to serverless compute, which runs on Spark Connect. Which statement correctly describes how the two architectures differ in handling the job's queries?
Both architectures run transformations lazily and compute when an action is called. The key difference is that Spark Connect defers analysis (name resolution) until the plan reaches the server, while classic Spark analyzes eagerly.
“The key difference between Spark Connect and Spark Classic is that Spark Connect defers analysis and name resolution to execution time”Source: docs.databricks.com
2.Unresolved plans out, Arrow batches back
Every DataFrame call on the client adds to an unresolved logical plan. The plan records what you asked for, such as a filter on amount or a projection of id and amount, without checking whether those columns exist. The client encodes the plan with Protocol Buffers and sends it to the server over gRPC. The plan is language-agnostic, which is why the same protocol can serve PySpark, Scala and other clients.
The server does the work a classic driver would do. Its Spark Connect endpoint turns the received plan into Spark's own logical plan operators, much as a SQL query is parsed into an initial plan. From there the normal analysis, optimization and execution pipeline runs, so a Connect query gets the same optimizations as a classic one. Results come back over the same gRPC channel as Apache Arrow-encoded row batches.
| Direction | Content | Encoding / transport |
|---|---|---|
| Client to server | Unresolved logical plan built from DataFrame API calls | Protocol Buffers over gRPC |
| Server to client | Query results | Apache Arrow-encoded row batches streamed through gRPC |
Checkpoint 2 of 10· Match them up
Match each piece of the Spark Connect protocol to the role it plays when a query travels between client and server.
Tap a term, then the definition that fits it.
The client turns DataFrame calls into an unresolved logical plan and encodes it with Protocol Buffers. It sends the plan over gRPC, and the server streams results back over the same gRPC channel as Arrow-encoded row batches.
“Results are streamed back to the client through gRPC as Apache Arrow-encoded row batches.”Source: spark.apache.org
Checkpoint 3 of 10· Exam question
The following PySpark code runs against a Spark Connect endpoint:
df = spark.read.table("sales")
df2 = df.filter(df.amount > 100).select("id", "amount")
Before df2 reaches the Spark Connect server, how is this chain of filter and select calls represented on the wire?
Correct answer: A — As an unresolved logical plan built from the DataFrame API calls, encoded with Protocol Buffers and sent over a gRPC connection to the server.
- A. Correct — Spark Connect clients build an unresolved logical plan from DataFrame calls and encode it with Protocol Buffers before sending it over gRPC; the server resolves and optimizes it on arrival.
- B. Incorrect — the Spark Connect client has no access to internal Java references such as
_jdf, and no Py4J socket is used at all; that direct-JVM bridge is exactly what Spark Connect removes. - C. Incorrect — the client never computes partition or shuffle assignments itself; physical planning and execution happen entirely on the server after it receives the unresolved plan.
- D. Incorrect — DataFrame operations are not converted to SQL text on the client, and Spark Connect communicates over gRPC rather than a JDBC/Thrift connection.
- E. Incorrect — no bytecode compilation or class loading occurs on the client; the transformation chain stays as a structured plan object until the server resolves it.
3.What the thin client gives up, and what it gains
A Spark Connect client runs in a different process from the Spark driver, so it cannot reach into the driver's JVM. In PySpark this means there is no Py4J bridge. Private fields that held JVM objects, such as df._jdf, are not available. Code that used these internals to change the execution environment breaks under Connect.
The protocol describes work only as logical plans, so APIs that do not fit that model are not supported. The most important of these is the RDD API. The client also gets a session of its own rather than control of the cluster. It cannot see the SparkContext or the static Spark configuration, because those affect every client connected to the same server.
The isolation is what solves several multi-tenant problems:
- Stability: an application that uses too much memory affects only its own process, and client-side dependencies cannot conflict with the driver's.
- Upgradability: the Spark driver can be upgraded independently of the applications, as long as the server-side RPC definitions stay backwards compatible.
- Debuggability and observability: you can debug interactively from an IDE and monitor the application with its own framework's metrics and logging.
The split also changes where your code runs. Plain Python or Scala logic runs on the client, which is why a breakpoint in your IDE works. DataFrame operations run on the server. Where the driver process itself is placed when you submit an application with spark-submit is a separate question about deploy modes.
Checkpoint 4 of 10· Check yourself
Before pointing an existing PySpark job at a Spark Connect server, an engineer reviews four lines from it. Which line can run unchanged under Spark Connect?
A groupBy with count and collect is plain DataFrame work, so it fits the logical-plan model and runs on the server. The other lines need things the Connect client cannot reach: the SparkContext, the RDD API and the driver JVM through Py4J (df._jdf).
“the Spark Connect protocol uses Sparks logical plans as the abstraction to be able to declaratively describe the operations to be executed on the server.”Source: spark.apache.org
4.Opening a remote session with sc://
A Spark Connect session starts from a URL with the sc:// scheme that points at the server. Under Connect, the SparkSession lifecycle is split in two. Your process holds a lightweight handle, and the session itself lives on the server. In code you pass the URL to SparkSession.builder.remote(...). With no port, as in sc://localhost, the client uses the default Spark Connect port, 15002.
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost").getOrCreate()You can also connect without changing any code. If the SPARK_REMOTE environment variable is set on the client machine, for example export SPARK_REMOTE="sc://localhost", a newly created session becomes a Spark Connect session. The shells accept the URL as a flag: ./bin/pyspark --remote "sc://localhost" or ./bin/spark-shell --remote "sc://localhost". The URL can carry a host, a port and parameters, as in sc://myhost.com:443/;token=ABCDEFG. A session created without any of these mechanisms is a classic session.
On the client side, a Python application only needs the pyspark-client package and runs with the ordinary Python interpreter. Scala projects add the spark-connect-client-jvm dependency. The server is started with ./sbin/start-connect-server.sh and should be the same Spark version as the client package.
Run type(spark). A Spark Connect session reports <class 'pyspark.sql.connect.session.SparkSession'>, and the .connect. in the class path is the giveaway. The shell's welcome message also says 'Client connected to the Spark Connect server at …'.
Checkpoint 5 of 10· Fill the gap
This PySpark code is meant to open a Spark Connect session against a local server. Which builder method fills the blank?
from pyspark.sql import SparkSession
spark = SparkSession.builder. ? ("sc://localhost").getOrCreate()SparkSession.builder.remote() takes the sc:// URL of the Spark Connect server, and the session that results is a Spark Connect session. master() sets the cluster URL for a classic session. config() sets individual options. The builder has no connect() method.
Source: spark.apache.org5.Deferred analysis: errors surface at the action
Classic Spark analyzes each step as you build it, so a bad column name fails right on the transformation line. Spark Connect builds unresolved plans on the client and defers their analysis. The plan goes to the server for analysis only when an operation needs a resolved plan: accessing a schema, explaining the plan, persisting a DataFrame or running an action. That is when the missing-column error appears.
This catches code that wraps transformations in try/except to handle analysis errors. On a classic cluster the except branch runs when a column name has a typo. Under Spark Connect the same block finishes without an error, and the exception appears later at show(), outside the handler. To keep that error handling, force analysis inside the try block with df.columns, df.schema or an action.
try:
df = ...
df.columns # This will trigger eager analysis
except Exception as e:
print(f"Error: {repr(e)}")Schema access costs a round trip. Classic Spark reads df.columns, df.schema and df.isStreaming locally. Under Connect each one triggers an analysis RPC, and the schema is cached after the first access. A loop that builds hundreds of new DataFrames and reads each one's schema turns into hundreds of server requests, so you get better performance by avoiding analysis requests on large numbers of DataFrames.
Checkpoint 6 of 10· Fill the gap
Under Spark Connect, this handler should catch a misspelled column name while still inside the try block. Which DataFrame attribute fills the blank?
try:
df = ...
df. ? # This will trigger eager analysis
except Exception as e:
print(f"Error: {repr(e)}")Under Spark Connect, schema access such as df.columns needs a resolved plan. It sends the plan to the server for analysis, so a missing-column error is raised inside the try block. limit, alias and filter are lazy transformations that only extend the unresolved plan on the client, so nothing is raised inside the try block.
Source: docs.databricks.comSources1
6.Views and UDFs are bound when the query runs
Deferred analysis covers the session state a DataFrame depends on, not just column names. Under Spark Connect, temporary views, UDFs and configs that a DataFrame uses are evaluated when its plan executes. In classic Spark they are captured when the DataFrame is created. Two common patterns behave differently as a result.
Temporary views are looked up by name. A Connect DataFrame built from spark.table("v") stores only the name v. If you later call createOrReplaceTempView("v") with different data, the existing DataFrame returns the new data the next time it runs. Classic Spark embeds the view's plan in the DataFrame at creation, so replacing the view does not affect it. The fix is to give each view a unique name, for example by including a UUID. Creating a temporary view is still analyzed eagerly under Connect, so the view's own dependencies are fixed when you create it.
import uuid
def create_temp_view_and_create_dataframe(x):
temp_view_name = f"`temp_view_{uuid.uuid4()}`" # Use a random name to avoid conflicts.
spark.range(x).createOrReplaceTempView(temp_view_name)
return spark.table(temp_view_name)
df10 = create_temp_view_and_create_dataframe(10)
assert len(df10.collect()) == 10
df100 = create_temp_view_and_create_dataframe(100)
assert len(df10.collect()) == 10 # It works as expected now.
assert len(df100.collect()) == 100Python UDFs are serialized when the query executes, not when the UDF is defined. A UDF that reads an outer variable sees whatever value the variable holds at the action. In the sample below, classic Spark captures x = 123 when the UDF is created and prints 123. Spark Connect serializes the function and uploads it at show(), so it prints 456.
from pyspark.sql.functions import udf
x = 123
@udf("INT")
def foo():
return x
df = spark.range(1).select(foo())
x = 456
df.show() # Prints 456To fix the value when the UDF is defined, build the UDF with a function factory. A helper such as make_udf(value) takes the current value as an argument and returns udf(foo) from inside its own scope, so each UDF it produces keeps its own copy. With foo_udf = make_udf(x) followed by x = 456, the query still prints 123.
Checkpoint 7 of 10· Fill the gap
Serverless compute runs this PySpark code over Spark Connect. Which value completes the comment on the last line, showing what df.show() prints?
from pyspark.sql.functions import udf
x = 123
@udf("INT")
def foo():
return x
df = spark.range(1).select(foo())
x = 456
df.show() # Prints ? Under Spark Connect, a Python UDF is serialized and uploaded only when the query runs at show(). By then x holds 456. Classic Spark captures the UDF when it is created, so it would print 123.
Source: docs.databricks.comSources1
7.Databricks Connect builds on Spark Connect
Databricks Connect is not a separate protocol. For Databricks Runtime 13.3 LTS and above, it extends open-source Spark Connect with changes that support Databricks compute modes and Unity Catalog. It uses the same unresolved logical plans and Arrow over gRPC, and it lets you connect to Databricks compute from VS Code, PyCharm, IntelliJ IDEA, notebooks or a custom application, in Python, R or Scala.
| Code | Runs on | Consequence |
|---|---|---|
| General Python or Scala code | Local client | Interactive debugging in the IDE works |
| DataFrame transformations | Databricks compute, via the remote session | Results reach the client on collect(), show(), toPandas() |
| UDFs, foreach, foreachBatch, transformWithState | Databricks compute, after serialization | UDF dependencies must be installed on Databricks |
| Application dependencies | Local client | Install them in the project, e.g. its Python virtual environment |
The dependency split in the last row trips people up. A library imported only by driver-side Python belongs in your local virtual environment. A library that a UDF imports has to exist on the Databricks compute, because that is where the serialized UDF runs.
Deploy mode is a different question from Connect, and it answers where the driver process runs when an application is submitted with spark-submit. In cluster mode, the framework launches the driver inside the cluster. In client mode, the submitter launches the driver outside the cluster. Spark Connect moves your application code off the driver altogether, while deploy mode only places the driver.
Checkpoint 8 of 10· Check yourself
Working in PyCharm with Databricks Connect, a developer writes a Python UDF that imports a third-party parsing library. The library is installed only in the project's local virtual environment. Where does the library need to be installed for the UDF to work?
With Databricks Connect, UDFs defined locally are serialized and sent to Databricks compute, which is where they run. That makes the compute the place where the UDF's dependencies must exist. The local virtual environment is only for application dependencies, which run on the client.
“UDF code runs on Databricks compute: UDFs defined locally are serialized and transmitted to the cluster where it runs.”Source: docs.databricks.com
Checkpoint 9 of 10· Exam question
How does Databricks Connect relate to open-source Spark Connect, and which Databricks Runtime versions support it?
Correct answer: A — Databricks Connect extends Spark Connect with additions for Databricks compute modes and Unity Catalog, and it requires Databricks Runtime 13.3 LTS or above.
- A. Correct — Databricks Connect is built as an extension of the open-source Spark Connect protocol, adding support for Databricks compute modes and Unity Catalog, and it requires Databricks Runtime 13.3 LTS or newer.
- B. Incorrect — Databricks Connect is explicitly built on top of the Spark Connect protocol rather than being unrelated to it, and its Unity Catalog extensions require a newer runtime, not an older one.
- C. Incorrect — Databricks Connect does not replace or remove Spark Connect; it layers Databricks-specific additions onto the same protocol and targets Databricks compute, not an external standalone cluster.
- D. Incorrect — Databricks Connect uses the gRPC-based Spark Connect protocol rather than the Py4J driver bridge, and it has specific minimum runtime requirements rather than working on every version.
- E. Incorrect — Databricks Connect can target multiple Databricks compute types, including all-purpose and job clusters as well as SQL warehouses, so this restriction does not describe it.
Checkpoint 10 of 10· Exam question
A data engineering team submits a nightly ETL job as a scheduled Databricks job with no interactive user present, and needs the driver process to run inside the cluster so it keeps running even if the submitting machine goes offline:
spark-submit \
--deploy-mode ____ \
--class com.acme.etl.NightlyJob \
etl-job.jar
Which value correctly fills the blank?
Correct answer: A — cluster
- A. Correct — cluster deploy mode launches the driver process inside the cluster itself, so the job keeps running independently of whether the submitting machine stays online.
- B. Incorrect — client deploy mode launches the driver on the submitting machine, so the job would fail or stop if that machine goes offline before completion.
- C. Incorrect — local is not a
spark-submitdeploy mode value; it refers to running Spark entirely in a single JVM process without a cluster at all. - D. Incorrect — standalone describes a cluster manager type, not a deploy mode value accepted by the
--deploy-modeflag. - E. Incorrect — embedded is not a recognized
spark-submitdeploy mode; only client and cluster are valid values for this flag.
Sources
(5)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Spark Connect defers analysis and name resolution to execution time”
↩︎ A decoupled client for a remote Spark server“Python notebooks with Databricks Runtime version 14.3 and above, on standard compute”
↩︎ A decoupled client for a remote Spark server“accessing a schema, explaining the plan, persisting a DataFrame, or executing an action”
↩︎ Deferred analysis: errors surface at the action“Performance can be improved if you avoid analysis requests on large numbers of DataFrames.”
↩︎ Deferred analysis: errors surface at the action“To mitigate the difference, always create unique temporary view names.”
↩︎ Views and UDFs are bound when the query runs“The analysis is triggered eagerly when creating the temporary view”
↩︎ Views and UDFs are bound when the query runs“In Spark Connect, Python UDFs are lazy. Their serialization and registration are deferred until execution time.”
↩︎ Views and UDFs are bound when the query runs“use a function factory (closure with early binding) to correctly capture variable values.”
↩︎ Views and UDFs are bound when the query runs“Spark Connect is a gRPC-based protocol within Apache Spark that specifies how a client application can communicate with a remote Spark Server.”
↩︎ Key concept“in Spark Connect, this code does not cause any issue, as it only constructs a local unresolved plan without triggering any analysis.”
↩︎ Exam trap 2“In Spark Connect, the DataFrame stores only a reference to the temporary view by name.”
↩︎ Exam trap 3“The key difference between Spark Connect and Spark Classic is that Spark Connect defers analysis and name resolution to execution time”
↩︎ Checkpoint“will not throw any error because the unresolved plan is client-side only, but on df.columns or df.show() an error will be thrown”
↩︎ Prediction - 2.https://spark.apache.org/docs/latest/spark-connect-overview.htmlSecondary source
“In Apache Spark 3.4, Spark Connect introduced a decoupled client-server architecture”
↩︎ A decoupled client for a remote Spark server“The Spark Connect client translates DataFrame operations into unresolved logical query plans which are encoded using protocol buffers.”
↩︎ Unresolved plans out, Arrow batches back“Results are streamed back to the client through gRPC as Apache Arrow-encoded row batches.”
↩︎ Unresolved plans out, Arrow batches back“the Spark Connect protocol does not support all the execution APIs of Spark, most importantly RDDs.”
↩︎ What the thin client gives up, and what it gains“Most importantly, the client does not have access to the static Spark configuration or the SparkContext.”
↩︎ What the thin client gives up, and what it gains“The Spark driver can now seamlessly be upgraded independently of applications”
↩︎ What the thin client gives up, and what it gains“By default, the REPL will attempt to connect to a local Spark Server on port 15002.”
↩︎ Opening a remote session with sc://“If you set the SPARK_REMOTE environment variable on the client machine where your Spark client application is running”
↩︎ Opening a remote session with sc://“the Spark Connect protocol uses Sparks logical plans as the abstraction to be able to declaratively describe the operations to be executed on the server.”
↩︎ Checkpoint - 3.
“The underlying protocol uses Spark unresolved logical plans and Apache Arrow on top of gRPC.”
↩︎ Unresolved plans out, Arrow batches back“General code runs locally: Python and Scala code runs on the client side, enabling interactive debugging.”
↩︎ What the thin client gives up, and what it gains“Databricks Connect is an extension of Spark Connect with additions and modifications to support working with Databricks compute modes and Unity Catalog.”
↩︎ Databricks Connect builds on Spark Connect“UDFs defined locally are serialized and transmitted to the cluster where it runs.”
↩︎ Databricks Connect builds on Spark Connect“The underlying protocol uses Spark unresolved logical plans and Apache Arrow on top of gRPC.”
↩︎ Exam trap 1“Databricks Connect is an extension of Spark Connect with additions and modifications to support working with Databricks compute modes and Unity Catalog.”
↩︎ Exam trap 4“UDF code runs on Databricks compute: UDFs defined locally are serialized and transmitted to the cluster where it runs.”
↩︎ Checkpoint - 4.https://docs.databricks.com/aws/en/pyspark/reference/classes/sparksession/builder/remoteOfficial docs
“Sets the Spark remote URL to connect to, such as "sc://host:port" to run via a Spark Connect server.”
↩︎ Opening a remote session with sc:// - 5.https://spark.apache.org/docs/latest/cluster-overview.htmlSecondary source
“In "cluster" mode, the framework launches the driver inside of the cluster. In "client" mode, the submitter launches the driver outside of the cluster.”
↩︎ Databricks Connect builds on Spark Connect