Spark Connect vs Spark Classic: Lazy Analysis, Temp Views, UDFs and Databricks Connect

1.Where Spark Connect runs, and Databricks Connect

Spark Connect is an open-source, gRPC-based protocol in Apache Spark. It lets a client run Spark workloads remotely through the DataFrame API. On Databricks it powers more surfaces than many candidates expect:

- Scala notebooks on Databricks Runtime 13.3 and above, on standard compute - Python notebooks on Databricks Runtime 14.3 and above, on standard compute - serverless compute - Databricks Connect - Lakeflow pipelines with environment versions configured

Databricks Connect is the client library for connecting IDEs, notebooks and custom applications to Databricks compute. For Databricks Runtime 13.3 LTS and above, it is an extension of Spark Connect, with additions and modifications to support Databricks compute modes and Unity Catalog. You create sessions with DatabricksSession instead of SparkSession. Databricks Connect can also run against an open-source Spark Connect server, but code that uses Databricks-only features may fail there.

For exam questions, the most useful thing to know is the split between work done locally and work done remotely:

- General code runs locally. Your Python or Scala code runs on the client, so you can debug it interactively. - DataFrame operations run on Databricks compute. Transformations become Spark plans that run remotely. Results only come back to the client when you call collect(), show() or toPandas(). - UDF code also runs on Databricks compute. UDFs you define locally are serialized and sent to the cluster. So application dependencies are installed locally, but UDF dependencies have to be installed on Databricks.

Sources123

2.Deferred analysis: errors appear later than you expect

Both Spark Classic and Spark Connect execute queries lazily. Transformations and spark.sql SELECTs build a plan, while actions and SQL commands such as INSERT or CREATE run eagerly. The key difference is analysis. Spark Connect defers analysis and name resolution until execution time.

Spark Classic resolves the plan while you build it, so a misspelled column fails on the line where you wrote it. A Spark Connect client keeps an unresolved plan locally. Anything that needs a resolved plan sends it to the server over RPC: accessing a schema, explaining the plan, persisting a DataFrame, or running an action. The server analyses the plan only at that point.

This has two practical consequences. First, a try/except around a chain of transformations catches nothing in Spark Connect. If you depend on that exception, force analysis inside the try block with df.columns, df.schema or df.collect(). Second, schema access is no longer free. The first access to a DataFrame's schema triggers an RPC, which then caches the schema, so you get better performance by avoiding analysis requests on large numbers of DataFrames.

try:
  df = ...
  df. ?  # This will trigger eager analysis
except Exception as e:
  print(f"Error: {repr(e)}")

Sources2

3.Temp views and UDFs bind at execution time

Deferred resolution also changes what a DataFrame refers to.

Temporary views. In Spark Connect, a DataFrame holds only a reference to a temporary view by name. If you replace that view later, the DataFrame's data changes too, because the name is looked up at execution time. Spark Classic embeds the view's plan when the DataFrame is created, so replacing the view has no effect on it. The documented fix is to always create unique temporary view names, for example by including a UUID.

Python UDFs. In Spark Connect, Python UDFs are lazy. They are serialized and registered only at execution time. A UDF that reads an outer variable sees the variable's value when the action runs, not when the UDF was defined. Spark Classic captures the value at creation.

from pyspark.sql.functions import udf

x = 123

@udf("INT")
def foo():
  return x


df = spark.range(1).select(foo())
x = 456
df.show() # Prints 456

If a UDF has to depend on an external value, bind that value early with a function factory. Wrap the UDF creation in a helper function and pass in the current value as an argument, so each generated UDF keeps its own copy.

from pyspark.sql.functions import udf

def make_udf(value):
  def foo():
    return value
  return udf(foo)


x = 123
foo_udf = make_udf(x)
x = 456
df = spark.range(1).select(foo_udf())
df.show() # Prints 123 as expected

Sources2

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.In Spark Connect, referencing a non-existent column in filter() raises an analysis error on that line, just as in Spark Classic.Why is that wrong?

   Spark Connect only builds an unresolved plan on the client. The error appears when schema access or an action sends the plan to the server for analysis.

   Covered in Deferred analysis: errors appear later than you expect

2. 2.A Python UDF captures the values of outer variables at the moment it is defined, whichever Spark mode you use.Why is that wrong?

   That is true only in Spark Classic. In Spark Connect, Python UDFs are serialized at execution time, so they see the latest values unless you bind them with a function factory.

   Covered in Temp views and UDFs bind at execution time

3. 3.Databricks Connect is a separate protocol that replaces Spark Connect.Why is that wrong?

   For Databricks Runtime 13.3 LTS and above, Databricks Connect is built on open-source Spark Connect and extends it for Databricks compute modes and Unity Catalog.

   Covered in Where Spark Connect runs, and Databricks Connect

Practise it for real

Watch Spark Connect defer analysis by running a broken query against a local Spark Connect server

1. 1.From the extracted Spark folder, run ./sbin/start-connect-server.sh

   Why: This starts a Spark server with the Spark Connect endpoint enabled

   You should see: The server is running and ready to accept Spark Connect sessions

2. 2.Run export SPARK_REMOTE="sc://localhost" and then start ./bin/pyspark

   Why: Setting SPARK_REMOTE turns the shell's session into a Spark Connect session without any code change

   You should see: The welcome message says: Client connected to the Spark Connect server at localhost

3. 3.Run type(spark)

   Why: This confirms which kind of session you have

   You should see: <class 'pyspark.sql.connect.session.SparkSession'>, whose path includes .connect.

4. 4.Run df = spark.sql("select 1 as a, 2 as b").filter("c > 1")

   Why: The client only builds an unresolved plan locally

   You should see: No error, even though column c doesn't exist

5. 5.Run df.columns

   Why: Schema access sends the plan to the server for analysis

   You should see: An error saying column c can't be found