CertSafari
    Databricks Certified Associate Developer for Apache Spark· Lessons

    Domain 6 · Lesson 29/32

    Spark Connect: thin-client architecture and deferred analysis

    Describe the features of Spark Connect.

    19 min read
    2.5% of exam
    5 sources
    Published 21 Oct 2026
    Docs as of 30 Sep 2026

    What you will be able to do

    • Describe how a Spark Connect client encodes DataFrame operations and how results come back
    • Explain which Spark APIs a Spark Connect client cannot reach and what the separation buys
    • Create a remote session with sc:// through SparkSession.builder.remote, SPARK_REMOTE or --remote
    • Predict when an analysis error, a replaced temporary view or a changed UDF variable takes effect under Spark Connect
    • State how Databricks Connect relates to Spark Connect and which runtimes it covers

    Key concept

    Spark Connect — A protocol that splits Spark into a thin client and a remote server. The client only builds unresolved plans and sends them over gRPC. The server analyzes them, executes them and streams the results back, so nothing is resolved until the server sees the plan.

    1.A decoupled client for a remote Spark server

    Spark Connect separates the code that describes a query from the process that runs it. Apache Spark 3.4 introduced it as a decoupled client-server architecture: your application holds a small client library, and a remote Spark server holds the driver. The client can be embedded almost anywhere, including application servers, IDEs, notebooks and other programming languages, because it no longer has to host Spark itself.

    The term usually comes up on Databricks, because many Databricks environments already run on Spark Connect. Python notebooks on standard compute with Databricks Runtime 14.3 and above use it, as do Scala notebooks on 13.3 and above, serverless compute, Databricks Connect and Lakeflow pipelines with environment versions configured. Code that ran fine on a classic cluster can behave differently once it lands on one of these, so the differences in this lesson matter even if you never start a Connect server yourself.

    Classic Spark and Spark Connect agree on one thing. Both run transformations lazily, and an action such as show() or collect() triggers the computation. They differ in when a query is analyzed: when column and table names are resolved and checked. Most of what is surprising about Spark Connect follows from that one difference.

    Checkpoint 1 of 10· Check yourself

    A team moves a PySpark job from a classic cluster to serverless compute, which runs on Spark Connect. Which statement correctly describes how the two architectures differ in handling the job's queries?

    Sources12

    2.Unresolved plans out, Arrow batches back

    Every DataFrame call on the client adds to an unresolved logical plan. The plan records what you asked for, such as a filter on amount or a projection of id and amount, without checking whether those columns exist. The client encodes the plan with Protocol Buffers and sends it to the server over gRPC. The plan is language-agnostic, which is why the same protocol can serve PySpark, Scala and other clients.

    The server does the work a classic driver would do. Its Spark Connect endpoint turns the received plan into Spark's own logical plan operators, much as a SQL query is parsed into an initial plan. From there the normal analysis, optimization and execution pipeline runs, so a Connect query gets the same optimizations as a classic one. Results come back over the same gRPC channel as Apache Arrow-encoded row batches.

    What crosses the wire in each direction
    DirectionContentEncoding / transport
    Client to serverUnresolved logical plan built from DataFrame API callsProtocol Buffers over gRPC
    Server to clientQuery resultsApache Arrow-encoded row batches streamed through gRPC

    Checkpoint 2 of 10· Match them up

    Match each piece of the Spark Connect protocol to the role it plays when a query travels between client and server.

    Tap a term, then the definition that fits it.

    Checkpoint 3 of 10· Exam question

    The following PySpark code runs against a Spark Connect endpoint: df = spark.read.table("sales") df2 = df.filter(df.amount > 100).select("id", "amount") Before df2 reaches the Spark Connect server, how is this chain of filter and select calls represented on the wire?

    Sources32

    3.What the thin client gives up, and what it gains

    A Spark Connect client runs in a different process from the Spark driver, so it cannot reach into the driver's JVM. In PySpark this means there is no Py4J bridge. Private fields that held JVM objects, such as df._jdf, are not available. Code that used these internals to change the execution environment breaks under Connect.

    The protocol describes work only as logical plans, so APIs that do not fit that model are not supported. The most important of these is the RDD API. The client also gets a session of its own rather than control of the cluster. It cannot see the SparkContext or the static Spark configuration, because those affect every client connected to the same server.

    The isolation is what solves several multi-tenant problems:

    • Stability: an application that uses too much memory affects only its own process, and client-side dependencies cannot conflict with the driver's.
    • Upgradability: the Spark driver can be upgraded independently of the applications, as long as the server-side RPC definitions stay backwards compatible.
    • Debuggability and observability: you can debug interactively from an IDE and monitor the application with its own framework's metrics and logging.

    The split also changes where your code runs. Plain Python or Scala logic runs on the client, which is why a breakpoint in your IDE works. DataFrame operations run on the server. Where the driver process itself is placed when you submit an application with spark-submit is a separate question about deploy modes.

    Checkpoint 4 of 10· Check yourself

    Before pointing an existing PySpark job at a Spark Connect server, an engineer reviews four lines from it. Which line can run unchanged under Spark Connect?

    Sources23

    4.Opening a remote session with sc://

    A Spark Connect session starts from a URL with the sc:// scheme that points at the server. Under Connect, the SparkSession lifecycle is split in two. Your process holds a lightweight handle, and the session itself lives on the server. In code you pass the URL to SparkSession.builder.remote(...). With no port, as in sc://localhost, the client uses the default Spark Connect port, 15002.

    Creating a Spark Connect session in PySparkpython
    from pyspark.sql import SparkSession
    spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

    You can also connect without changing any code. If the SPARK_REMOTE environment variable is set on the client machine, for example export SPARK_REMOTE="sc://localhost", a newly created session becomes a Spark Connect session. The shells accept the URL as a flag: ./bin/pyspark --remote "sc://localhost" or ./bin/spark-shell --remote "sc://localhost". The URL can carry a host, a port and parameters, as in sc://myhost.com:443/;token=ABCDEFG. A session created without any of these mechanisms is a classic session.

    On the client side, a Python application only needs the pyspark-client package and runs with the ordinary Python interpreter. Scala projects add the spark-connect-client-jvm dependency. The server is started with ./sbin/start-connect-server.sh and should be the same Spark version as the client package.

    Checkpoint 5 of 10· Fill the gap

    This PySpark code is meant to open a Spark Connect session against a local server. Which builder method fills the blank?

    from pyspark.sql import SparkSession
    spark = SparkSession.builder. ? ("sc://localhost").getOrCreate()

    Sources42

    5.Deferred analysis: errors surface at the action

    Classic Spark analyzes each step as you build it, so a bad column name fails right on the transformation line. Spark Connect builds unresolved plans on the client and defers their analysis. The plan goes to the server for analysis only when an operation needs a resolved plan: accessing a schema, explaining the plan, persisting a DataFrame or running an action. That is when the missing-column error appears.

    This catches code that wraps transformations in try/except to handle analysis errors. On a classic cluster the except branch runs when a column name has a typo. Under Spark Connect the same block finishes without an error, and the exception appears later at show(), outside the handler. To keep that error handling, force analysis inside the try block with df.columns, df.schema or an action.

    Forcing analysis inside the try block so the exception is caughtpython
    try:
      df = ...
      df.columns # This will trigger eager analysis
    except Exception as e:
      print(f"Error: {repr(e)}")

    Schema access costs a round trip. Classic Spark reads df.columns, df.schema and df.isStreaming locally. Under Connect each one triggers an analysis RPC, and the schema is cached after the first access. A loop that builds hundreds of new DataFrames and reads each one's schema turns into hundreds of server requests, so you get better performance by avoiding analysis requests on large numbers of DataFrames.

    Checkpoint 6 of 10· Fill the gap

    Under Spark Connect, this handler should catch a misspelled column name while still inside the try block. Which DataFrame attribute fills the blank?

    try:
      df = ...
      df. ?  # This will trigger eager analysis
    except Exception as e:
      print(f"Error: {repr(e)}")

    Sources1

    6.Views and UDFs are bound when the query runs

    Deferred analysis covers the session state a DataFrame depends on, not just column names. Under Spark Connect, temporary views, UDFs and configs that a DataFrame uses are evaluated when its plan executes. In classic Spark they are captured when the DataFrame is created. Two common patterns behave differently as a result.

    Temporary views are looked up by name. A Connect DataFrame built from spark.table("v") stores only the name v. If you later call createOrReplaceTempView("v") with different data, the existing DataFrame returns the new data the next time it runs. Classic Spark embeds the view's plan in the DataFrame at creation, so replacing the view does not affect it. The fix is to give each view a unique name, for example by including a UUID. Creating a temporary view is still analyzed eagerly under Connect, so the view's own dependencies are fixed when you create it.

    Unique view names keep earlier DataFrames stable under Spark Connectpython
    import uuid
    def create_temp_view_and_create_dataframe(x):
      temp_view_name = f"`temp_view_{uuid.uuid4()}`"  # Use a random name to avoid conflicts.
      spark.range(x).createOrReplaceTempView(temp_view_name)
      return spark.table(temp_view_name)
    
    df10 = create_temp_view_and_create_dataframe(10)
    assert len(df10.collect()) == 10
    
    df100 = create_temp_view_and_create_dataframe(100)
    assert len(df10.collect()) == 10  # It works as expected now.
    assert len(df100.collect()) == 100

    Python UDFs are serialized when the query executes, not when the UDF is defined. A UDF that reads an outer variable sees whatever value the variable holds at the action. In the sample below, classic Spark captures x = 123 when the UDF is created and prints 123. Spark Connect serializes the function and uploads it at show(), so it prints 456.

    Under Spark Connect, the UDF picks up the variable's value at executionpython
    from pyspark.sql.functions import udf
    
    x = 123
    
    @udf("INT")
    def foo():
      return x
    
    
    df = spark.range(1).select(foo())
    x = 456
    df.show() # Prints 456

    To fix the value when the UDF is defined, build the UDF with a function factory. A helper such as make_udf(value) takes the current value as an argument and returns udf(foo) from inside its own scope, so each UDF it produces keeps its own copy. With foo_udf = make_udf(x) followed by x = 456, the query still prints 123.

    Checkpoint 7 of 10· Fill the gap

    Serverless compute runs this PySpark code over Spark Connect. Which value completes the comment on the last line, showing what df.show() prints?

    from pyspark.sql.functions import udf
    
    x = 123
    
    @udf("INT")
    def foo():
      return x
    
    
    df = spark.range(1).select(foo())
    x = 456
    df.show() # Prints  ? 

    Sources1

    7.Databricks Connect builds on Spark Connect

    Databricks Connect is not a separate protocol. For Databricks Runtime 13.3 LTS and above, it extends open-source Spark Connect with changes that support Databricks compute modes and Unity Catalog. It uses the same unresolved logical plans and Arrow over gRPC, and it lets you connect to Databricks compute from VS Code, PyCharm, IntelliJ IDEA, notebooks or a custom application, in Python, R or Scala.

    Where each kind of code runs with Databricks Connect
    CodeRuns onConsequence
    General Python or Scala codeLocal clientInteractive debugging in the IDE works
    DataFrame transformationsDatabricks compute, via the remote sessionResults reach the client on collect(), show(), toPandas()
    UDFs, foreach, foreachBatch, transformWithStateDatabricks compute, after serializationUDF dependencies must be installed on Databricks
    Application dependenciesLocal clientInstall them in the project, e.g. its Python virtual environment

    The dependency split in the last row trips people up. A library imported only by driver-side Python belongs in your local virtual environment. A library that a UDF imports has to exist on the Databricks compute, because that is where the serialized UDF runs.

    Deploy mode is a different question from Connect, and it answers where the driver process runs when an application is submitted with spark-submit. In cluster mode, the framework launches the driver inside the cluster. In client mode, the submitter launches the driver outside the cluster. Spark Connect moves your application code off the driver altogether, while deploy mode only places the driver.

    Checkpoint 8 of 10· Check yourself

    Working in PyCharm with Databricks Connect, a developer writes a Python UDF that imports a third-party parsing library. The library is installed only in the project's local virtual environment. Where does the library need to be installed for the UDF to work?

    Checkpoint 9 of 10· Exam question

    How does Databricks Connect relate to open-source Spark Connect, and which Databricks Runtime versions support it?

    Checkpoint 10 of 10· Exam question

    A data engineering team submits a nightly ETL job as a scheduled Databricks job with no interactive user present, and needs the driver process to run inside the cluster so it keeps running even if the submitting machine goes offline: spark-submit \ --deploy-mode ____ \ --class com.acme.etl.NightlyJob \ etl-job.jar Which value correctly fills the blank?

    Sources35

    Sources

    (5)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Spark Connect defers analysis and name resolution to execution time”
      ↩︎ A decoupled client for a remote Spark server
      “Python notebooks with Databricks Runtime version 14.3 and above, on standard compute”
      ↩︎ A decoupled client for a remote Spark server
      “accessing a schema, explaining the plan, persisting a DataFrame, or executing an action”
      ↩︎ Deferred analysis: errors surface at the action
      “Performance can be improved if you avoid analysis requests on large numbers of DataFrames.”
      ↩︎ Deferred analysis: errors surface at the action
      “To mitigate the difference, always create unique temporary view names.”
      ↩︎ Views and UDFs are bound when the query runs
      “The analysis is triggered eagerly when creating the temporary view”
      ↩︎ Views and UDFs are bound when the query runs
      “In Spark Connect, Python UDFs are lazy. Their serialization and registration are deferred until execution time.”
      ↩︎ Views and UDFs are bound when the query runs
      “use a function factory (closure with early binding) to correctly capture variable values.”
      ↩︎ Views and UDFs are bound when the query runs
      “Spark Connect is a gRPC-based protocol within Apache Spark that specifies how a client application can communicate with a remote Spark Server.”
      ↩︎ Key concept
      “in Spark Connect, this code does not cause any issue, as it only constructs a local unresolved plan without triggering any analysis.”
      ↩︎ Exam trap 2
      “In Spark Connect, the DataFrame stores only a reference to the temporary view by name.”
      ↩︎ Exam trap 3
      “The key difference between Spark Connect and Spark Classic is that Spark Connect defers analysis and name resolution to execution time”
      ↩︎ Checkpoint
      “will not throw any error because the unresolved plan is client-side only, but on df.columns or df.show() an error will be thrown”
      ↩︎ Prediction
    2. 2.
      “In Apache Spark 3.4, Spark Connect introduced a decoupled client-server architecture”
      ↩︎ A decoupled client for a remote Spark server
      “The Spark Connect client translates DataFrame operations into unresolved logical query plans which are encoded using protocol buffers.”
      ↩︎ Unresolved plans out, Arrow batches back
      “Results are streamed back to the client through gRPC as Apache Arrow-encoded row batches.”
      ↩︎ Unresolved plans out, Arrow batches back
      “the Spark Connect protocol does not support all the execution APIs of Spark, most importantly RDDs.”
      ↩︎ What the thin client gives up, and what it gains
      “Most importantly, the client does not have access to the static Spark configuration or the SparkContext.”
      ↩︎ What the thin client gives up, and what it gains
      “The Spark driver can now seamlessly be upgraded independently of applications”
      ↩︎ What the thin client gives up, and what it gains
      “By default, the REPL will attempt to connect to a local Spark Server on port 15002.”
      ↩︎ Opening a remote session with sc://
      “If you set the SPARK_REMOTE environment variable on the client machine where your Spark client application is running”
      ↩︎ Opening a remote session with sc://
      “the Spark Connect protocol uses Sparks logical plans as the abstraction to be able to declaratively describe the operations to be executed on the server.”
      ↩︎ Checkpoint
    3. 3.
      “The underlying protocol uses Spark unresolved logical plans and Apache Arrow on top of gRPC.”
      ↩︎ Unresolved plans out, Arrow batches back
      “General code runs locally: Python and Scala code runs on the client side, enabling interactive debugging.”
      ↩︎ What the thin client gives up, and what it gains
      “Databricks Connect is an extension of Spark Connect with additions and modifications to support working with Databricks compute modes and Unity Catalog.”
      ↩︎ Databricks Connect builds on Spark Connect
      “UDFs defined locally are serialized and transmitted to the cluster where it runs.”
      ↩︎ Databricks Connect builds on Spark Connect
      “The underlying protocol uses Spark unresolved logical plans and Apache Arrow on top of gRPC.”
      ↩︎ Exam trap 1
      “Databricks Connect is an extension of Spark Connect with additions and modifications to support working with Databricks compute modes and Unity Catalog.”
      ↩︎ Exam trap 4
      “UDF code runs on Databricks compute: UDFs defined locally are serialized and transmitted to the cluster where it runs.”
      ↩︎ Checkpoint
    4. 4.
      “Sets the Spark remote URL to connect to, such as "sc://host:port" to run via a Spark Connect server.”
      ↩︎ Opening a remote session with sc://
    5. 5.
      “In "cluster" mode, the framework launches the driver inside of the cluster. In "client" mode, the submitter launches the driver outside of the cluster.”
      ↩︎ Databricks Connect builds on Spark Connect

    Ready to test yourself?

    Practise the 11 questions on this subdomain.

    Spotted a mistake, or was something unclear? Tell us.