Snowpark Architecture and Lazy DataFrames

1.Snowpark architecture: your code, Snowflake's compute

Snowpark is a client library for querying and processing data in Snowflake. Snowflake provides it in exactly three languages: Java, Python and Scala. Its main architectural idea is that the data never moves to your code. You write ordinary Java, Python or Scala on your laptop, in Jupyter, VS Code or IntelliJ. Snowpark turns that code into SQL, and the SQL runs inside Snowflake's engine. The documentation describes this as building applications "without moving data to the system where your application code runs".

The documentation compares Snowpark with the Snowflake Connector for Spark, and two benefits stand out for the exam. First, Snowpark offers pushdown for every operation, including Snowflake UDFs, so the heavy lifting happens where the data is stored. Second, Snowpark needs no separate cluster of its own. Snowflake does all the computation and also manages scaling and compute. So when a question asks where a Snowpark transformation executes, or what extra infrastructure you must provision, the answer is Snowflake itself and nothing extra.

The last row matters in day-to-day work. Snowpark gives you programming-language constructs for building SQL statements. Instead of putting a SELECT inside a string, you call methods and pass column objects. You can still run a string of SQL if you want to, but the native constructs give you code completion and type checking. Here is the Python version of a three-column projection:

>>> # Import the col function from the functions module.
>>> from snowflake.snowpark.functions import col

>>> # Create a DataFrame that contains the id, name, and serial_number
>>> # columns in the "sample_product_data" table.
>>> df = session.table("sample_product_data").select(col("id"), col("name"), col("serial_number"))
>>> df.show()

Sources1

2.Lazy evaluation: construct, transform, then act

Since the work happens on the server, Snowpark has to decide when to send it there. That raises a question about the line of code below.

The DataFrame is Snowpark's core abstraction. The Python, Java and Scala guides all describe the same three-step lifecycle. First, construct a DataFrame and name its source, such as a table, a staged file or a SQL statement. Second, describe the transformations: which columns to select, how to filter rows, and how to sort and group the results. Third, call a method that performs an action, such as collect(). Only the third step contacts the database.

The documentation gives the reason for this design. Snowpark operations run lazily on the server, so a pipeline can delay execution as long as possible while "batching up many operations into a single operation. This reduces the amount of data transferred between your client and the Snowflake database." One way to picture it: a DataFrame is like a query waiting to be evaluated.

>>> # Send the query to the server for execution and
>>> # return a list of Rows containing the results.
>>> results = df. ? ()

Sources12

3.Constructing a DataFrame from a Session

Every DataFrame starts from a Session, your connection to Snowflake. In Python, one documented approach is to put the connection parameters (account, user, role, warehouse, database, schema) in a dict and call Session.builder.configs(connection_parameters).create(). After that, the Session's methods and properties build DataFrames, and each one reads from a different kind of source.

Look closely at the read row. read is a property that returns a DataFrameReader, and you then call the method for the file's format, such as json or csv. You can also attach a schema first, built from StructType and StructField. You can likewise pass a StructType schema to create_dataframe so that local values get explicit column types.

session.sql can run SELECT statements against tables and staged files, but the documentation recommends something else. It says that "using the table method and read property offer better syntax highlighting, error highlighting, and intelligent code completion in development tools." The Java and Scala guides give the same advice. Two more details from those guides: in Java and Scala, the table method returns an Updatable object, which extends DataFrame with methods for updating and deleting table data. And in every language, words reserved by Snowflake are not valid as column names when you construct a DataFrame.

Sources324

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Snowpark, like the Spark connector, needs its own compute cluster next to Snowflake for heavy transformations.Why is that wrong?

   Snowpark pushes all operations down to Snowflake. No external cluster is involved, and Snowflake manages scale and compute.

   Covered in Snowpark architecture: your code, Snowflake's compute

2. 2.Calling session.table(...).select(...) runs a query and pulls the selected columns to the client.Why is that wrong?

   Constructing and transforming a DataFrame only describes the query. Snowflake runs SQL only when an action such as collect() is called.

   Covered in Lazy evaluation: construct, transform, then act

3. 3.session.sql("SELECT ...") is the recommended way to load a table or staged file into a DataFrame.Why is that wrong?

   sql works, but the docs recommend the table method and read property because they give better syntax highlighting, error highlighting and code completion.

   Covered in Constructing a DataFrame from a Session