What you will be able to do
- Explain why one engine for batch, SQL, streaming and ML is Spark's headline advantage when consolidating separate systems
- Explain why keeping intermediate data in memory speeds up iterative and multi-step jobs
- Describe how Spark recovers lost partitions through lineage, and how this differs from replication
- Name the built-in costs of running Spark: shuffles, spill and skew, the single driver, and isolated applications
- Recognise workloads where Spark's distribution overhead brings no benefit
Key concept
Lineage recomputation — Spark records the chain of transformations that produced each partition. When a partition is lost, Spark replays that chain to rebuild it, so it can tolerate failures without keeping a second copy of the data.
1.One engine for batch, SQL, streaming and machine learning
Spark's headline advantage is that one engine runs many kinds of work. A company that runs nightly ETL on one system, analyst SQL on a second and model training on a third has to move data between them and keep three platforms working. Spark replaces all three with a single stack: Spark SQL and DataFrames for structured processing, Structured Streaming for continuous data, and MLlib for machine learning. All of them run on the same cluster and the same execution engine.
The unification goes deeper than a shared installation. Spark SQL and the DataFrame and Dataset APIs all compile down to the same engine, whichever API or language expresses the computation. A developer can therefore switch between SQL and DataFrame code inside one pipeline without changing how it runs. Batch and streaming share the engine too: Spark and Structured Streaming have a unified architecture for both, and streaming can run either triggered or continuously, so each workload can trade cost against latency.
Machine learning belongs in the same picture. MLlib is optimised for large-scale machine learning algorithms and statistical modelling. Spark is especially strong at the data-heavy work around models: ETL, analytics reporting, feature engineering and data preprocessing. A demand-forecasting job can therefore read the tables that the nightly ETL just wrote, on the same cluster, with no hand-off to another system.
Checkpoint 1 of 8· Check yourself
A retailer runs nightly ETL on one system, analyst SQL on a second and demand-forecast model training on a third. Data has to be copied between them, and the team maintains three platforms. Which Spark advantage solves this problem directly?
Spark's main advantage here is a unified engine. ETL, SQL, streaming and ML all run on the same cluster and the same execution engine, so the forecasting job can read the ETL output directly. Spark does not turn jobs into MapReduce plans, ship a dashboard or create GPU pools.
“When computing a result, the same execution engine is used, independent of which API/language you are using to express the computation.”Source: spark.apache.org
Checkpoint 2 of 8· Exam question
A retail company runs separate Hadoop MapReduce jobs for nightly ETL, a dedicated SQL engine for analyst reporting, and a standalone cluster for demand-forecasting models, and wants to consolidate onto one platform without losing performance. Which advantage of Apache Spark most directly supports this consolidation?
Correct answer: A — Spark exposes a unified engine where Spark SQL, DataFrames, Structured Streaming, and MLlib share the same execution model, so ETL, reporting, and forecasting run on one cluster.
- A. This is correct: Spark's unified engine lets the same cluster and APIs handle batch ETL, SQL reporting, and MLlib model training, which is exactly the consolidation the company wants. This eliminates the need to operate three separate specialized systems.
- B. Spark does not translate jobs into MapReduce plans; it compiles work into its own DAG of stages and tasks executed by its own scheduler. Framing Spark as a MapReduce-compatibility layer misstates how its execution engine works.
- C. Spark does not enforce schema-on-write, and reading raw files does not remove the need for a query engine to serve reporting workloads. Schema inference and enforcement are configurable, not automatic replacements for a SQL layer.
- D. Spark itself has no built-in charting dashboard; visualization is handled by separate tools such as notebooks or BI products that connect to Spark. This option describes a capability Spark does not provide.
- E. Spark does not auto-provision GPU pools when ML code runs; hardware allocation depends on the cluster manager and the cluster configuration chosen by an administrator. This overstates what happens automatically.
2.In-memory processing and lazy planning
Spark keeps intermediate data in memory between steps instead of writing it out and reading it back. An executor is a process that runs tasks and keeps data in memory or on disk across them for the whole application. When a dataset is persisted, each node keeps the partitions it computed in memory and reuses them for later actions on that dataset or on datasets derived from it. Reuse is where the speedup comes from, often more than 10x. An iterative algorithm that passes over the same data twenty times reads it from storage once, not twenty times. That is why caching is described as a key tool for iterative algorithms and fast interactive use.
In-memory processing does not remove the ordering of work. Each job is split into stages that depend on each other, and a later stage still waits for the stage that produces its input. Memory is also finite. Spill happens when Spark runs low on execution memory, which it uses for shuffles, joins, sorts and aggregations. The data that does not fit is written to disk, adding disk I/O and extra garbage collection. Spark runs on the JVM, and its garbage collector is a cost to manage, not something Spark has engineered away.
Lazy evaluation adds a second advantage on top of memory. Transformations only add logic to a plan, and nothing runs until an action asks for a result (the mechanics are in Transformations, actions and lazy evaluation). An engine that runs each step as soon as it is written has to optimise one step at a time. Spark sees the whole plan first, so its optimisations can consider every transformation behind an action together and choose a plan based on how the data is physically laid out.
Checkpoint 3 of 8· Check yourself
A cluster was sized for in-memory processing. During a large join and aggregation, its executors still show heavy disk writes and longer garbage-collection pauses. What is the most likely explanation?
Spark uses execution memory for shuffles, joins, sorts and aggregations. When that memory runs low, it writes the data that does not fit to disk, which adds disk I/O and extra garbage collection. In-memory processing reduces disk use but does not remove it, and Spark does not turn jobs into MapReduce plans.
“When data does not fit in memory Spark will spill these tables to disk, incurring the additional overhead of disk I/O and increased garbage collection.”Source: spark.apache.org
Checkpoint 4 of 8· Exam question
An engineer migrates a nightly aggregation pipeline from disk-based MapReduce to Spark and observes a large runtime improvement on a job that repeatedly scans and re-scans the same intermediate dataset across several stages. Which characteristic of Spark's design explains this speedup?
Correct answer: A — Spark can hold intermediate results in executor memory across stages, avoiding the repeated disk reads and writes that MapReduce performs between each map and reduce phase.
- A. This is correct: Spark's in-memory processing lets it cache intermediate DataFrames or RDDs between stages, so a pipeline that reuses the same dataset avoids the round-trip to disk that MapReduce requires between phases. This is the core mechanism behind Spark's speed advantage on iterative or reused-data workloads.
- B. Spark does not automatically compress every dataset with a columnar codec between stages; compression is format- and configuration-dependent, not an automatic universal behavior. This is not the reason repeated scans get faster.
- C. Stages within a job generally have data dependencies, so later stages wait on the shuffle output of earlier ones rather than running fully in parallel. Describing all stages as simultaneous misrepresents Spark's DAG-based scheduling.
- D. Spark still runs on the JVM and is subject to its garbage collector; it does not replace GC with a pause-free custom manager. Memory tuning reduces GC impact but does not eliminate it.
- E. Spark does not compile job logic ahead of time into native machine code; execution still goes through the JVM (or Python worker processes for PySpark UDFs). Native compilation is not the mechanism behind the observed speedup.
3.Fault tolerance through lineage recomputation
When an executor is lost, Spark rebuilds the data rather than restoring a copy. Every partition has a recorded lineage, the transformations that produced it from its input. If a partition disappears with its executor, Spark replays those transformations on the original data and produces the partition again. This is why RDDs automatically recover from node failures, and why cached data is fault-tolerant without being stored twice.
Recovery has two parts: rebuilding the data and rescheduling the work. The driver schedules every task on the cluster. When an executor is removed, Spark reruns its unfinished tasks on the executors that remain. Consider a worker on a spot instance that the cloud provider reclaims. Its executors terminate and their active tasks fail. The job continues, and Spark reruns those tasks elsewhere. The result is still correct, but the job takes longer, and frequent losses can make a stage or the whole job fail.
| Event | What Spark does | Outcome |
|---|---|---|
| A worker spot instance is reclaimed | Its executors terminate, active tasks fail, and Spark reruns those tasks | The job continues but runs slower |
| Workers are lost repeatedly | Spark keeps rerunning the failed tasks | Stage or job failures become possible |
| The driver instance is reclaimed | There is no rerun, because the driver is the process that schedules tasks | The job fails immediately |
Replication is available, but only if you ask for it. Storage levels ending in _2, such as MEMORY_ONLY_2 and MEMORY_AND_DISK_2, keep each partition on two cluster nodes. With the default MEMORY_ONLY, a lost partition is recomputed. All storage levels give full fault tolerance through recomputation. The replicated levels only save the time spent waiting for a lost partition to be rebuilt, which matters for a low-latency case such as Spark serving requests from a web application.
| Storage level | Copies of each partition | When a partition is lost |
|---|---|---|
| MEMORY_ONLY (default for cache()) | One | Recomputed from its lineage |
| MEMORY_ONLY_2, MEMORY_AND_DISK_2 | Two cluster nodes | Tasks continue on the replica without waiting for recomputation |
Spark recomputed them. The driver rescheduled the lost tasks on surviving executors, and those tasks replayed the partitions' lineage from the source data. Spark does not drop lost partitions, return partial results, mirror partitions by default, or restore an executor's old memory.
Checkpoint 5 of 8· Match them up
A long-running Spark job hits each of these failures. Match each failure to what Spark does next.
Tap a term, then the definition that fits it.
Spark recovers lost data by recomputing it from lineage and reruns lost tasks on the executors that remain. A _2 storage level only saves the time a rebuild would take. The driver is different, because it schedules all the tasks. Nothing can rerun its work, so losing it fails the job.
“the replicated ones let you continue running tasks on the RDD without waiting to recompute a lost partition”Source: spark.apache.org
Checkpoint 6 of 8· Exam question
During a long-running Spark job, one executor is terminated after the underlying VM is reclaimed by the cloud provider. The job continues and eventually finishes with correct results. Which two statements correctly describe how Spark achieves this fault tolerance? (Choose 2 answers)(Select 2)
Correct answers: A, B — Spark tracks the lineage of transformations that produced each partition, so lost partitions on the failed executor can be recomputed from the original data and transformations.; The driver detects the missing executor's heartbeat and reschedules its pending and lost tasks onto other executors that remain part of the cluster.
- A. This is correct: Spark's RDD/DataFrame lineage graph records the sequence of transformations, so when a partition is lost it can be recomputed deterministically from source data rather than requiring a live replica. This lineage-based recovery is central to Spark's fault tolerance model.
- B. This is correct: the driver monitors executor heartbeats and, on detecting a failure, reassigns the affected tasks to healthy executors so the job can continue to completion. This rescheduling is what lets the job finish despite the lost VM.
- C. Spark does not silently drop lost partitions and return partial results; it recomputes them from lineage so the final output remains complete and correct. Returning incomplete data would defeat the purpose of fault tolerance.
- D. Spark does not synchronously mirror every partition to a second executor; that would double memory and network cost for every job. Recovery instead relies on recomputation from lineage, not live replication.
- E. A restarted executor process does not retain its prior in-memory state, since that state was lost when the VM was reclaimed. Spark must resubmit the lost tasks rather than assume the old state survives.
4.Challenges built into distributed processing
The shuffle is the main performance cost of distributed processing. Operations such as groupByKey, reduceByKey, join and repartition need all values for a key in one place, so Spark copies data across executors and machines. That involves disk I/O, data serialization and network I/O, and it also leaves intermediate files on disk. In a long-running application these files can take up a large amount of disk space. Shuffles cannot be avoided whenever data has to be regrouped, and much of Spark tuning comes down to shuffling less (Shuffles and wide transformations).
Memory pressure is the second built-in challenge. Shuffle operations build in-memory structures on the heap. When the data does not fit, Spark spills to disk and the stage slows down. Skew makes this worse. When one or a few tasks take much longer than the rest, the other cores sit idle until they finish, so a 200-core cluster can spend most of a stage waiting on one oversized partition. When executors run out of memory, Spark removes them and retries their tasks, and repeated failures make the stage and the job fail. Sizing memory per core is a real engineering task, covered in Executor and driver memory, spill and cluster sizing.
The design also concentrates risk in one process. Each application has a single driver that schedules its tasks, and executors connect back to it throughout the run (the roles are covered in Driver, worker nodes, executors and the cluster manager). Losing a worker costs time. Losing the driver ends the job. For this reason the driver should run on an on-demand instance even when the workers use spot capacity, and it should sit close to the workers on the network.
Isolation between applications brings a cost of its own. Each application gets its own executors, and tasks from different applications run in different JVMs. This keeps one team's job from disturbing another's. It also means two Spark applications cannot share an in-memory dataset. One must write the data to external storage before the other can read it.
Checkpoint 7 of 8· Check yourself
On a 200-core cluster, 199 tasks of a stage finish in two minutes. The last task, which reads one very large partition, runs for forty minutes. The team proposes doubling the number of workers. How much would that shorten this stage?
This is skew. One oversized partition holds up the whole stage while the other cores wait, so more workers only add idle cores. Spark does not split a running task across the new cores or across the cores that are already idle.
“create a few oversized tasks that hold up the whole stage”Source: docs.databricks.com
5.When Spark is the wrong tool
Spark pays off when the same operation runs over a large dataset. Its strength is data parallelism: split the data into partitions and apply the same logic to each partition at once. It is the recommended choice for large-scale joins, filters and aggregations. Everything else Spark does, including driver planning, task scheduling, shuffles and executors, is overhead that only a large dataset pays back.
On small data the overhead becomes the workload. Small input may fit in a single partition, so one task does all the work and extra workers do nothing. Splitting the data into many tiny tasks does not help either. Very small tasks have worse I/O throughput and suffer more from scheduling and task setup overhead. A workload that reads or changes only a few records falls into the same trap. A request for one record still becomes a job planned by the driver and run as tasks on executors, and with nothing to split across partitions, that machinery adds cost and buys no parallelism.
Some heavy workloads are the wrong shape for Spark. For task parallelism, many independent tasks running at once, Ray is the better fit. Reinforcement learning, simulation modelling, hyperparameter search and deep learning training are listed as workloads where Spark is less optimised. One common pattern uses Spark for data retrieval and preparation, then hands the compute-heavy stages to Ray running in the same environment.
| Workload | Better fit | Reason |
|---|---|---|
| ETL, analytics reporting, feature engineering on large tables | Spark | Data parallelism: the same operation applied to each element of a large dataset |
| Large-scale ML algorithms and statistical modelling | Spark (MLlib) | Optimised for large-scale machine learning |
| Hyperparameter search, simulation, deep learning training | Ray | Task parallelism: independent tasks run concurrently |
Checkpoint 8 of 8· Check yourself
Which plan fits a pipeline that builds features from large tables and then runs a hyperparameter search that trains hundreds of independent model variants?
Feature engineering on large tables is data parallelism, which is Spark's strength. A hyperparameter search is many independent tasks, which is task parallelism, and Ray is the better fit for it. The usual pattern is to prepare the data in Spark and pass the compute-heavy stage to Ray in the same environment.
“Use Spark for data-intensive tasks and switch to Ray for stages that require heavy computation.”Source: docs.databricks.com
Sources
(12)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“has a unified architecture for batch and streaming processing”
↩︎ One engine for batch, SQL, streaming and machine learning“Streaming processing can be run in both triggered and continuous manners”
↩︎ One engine for batch, SQL, streaming and machine learning - 2.https://spark.apache.org/docs/latest/sql-programming-guide.htmlSecondary source
“the same execution engine is used, independent of which API/language you are using to express the computation”
↩︎ One engine for batch, SQL, streaming and machine learning“When computing a result, the same execution engine is used, independent of which API/language you are using to express the computation.”
↩︎ Checkpoint - 3.
“Spark's MLlib and SparkML libraries are optimized for large-scale machine learning algorithms and statistical modeling.”
↩︎ One engine for batch, SQL, streaming and machine learning“It's ideal for ETL, analytics reporting, feature engineering, and data preprocessing.”
↩︎ One engine for batch, SQL, streaming and machine learning“Spark excels at data parallelism - Apply the same operation to each element of a large dataset.”
↩︎ When Spark is the wrong tool“Use Ray for workloads where Spark is less optimized”
↩︎ When Spark is the wrong tool“Use Spark for data-intensive tasks and switch to Ray for stages that require heavy computation.”
↩︎ When Spark is the wrong tool - 4.https://spark.apache.org/docs/latest/rdd-programming-guide.htmlSecondary source
“This nomenclature comes from MapReduce and does not directly relate to Spark’s map and reduce operations.”
↩︎ One engine for batch, SQL, streaming and machine learning“Caching is a key tool for iterative algorithms and fast interactive use.”
↩︎ In-memory processing and lazy planning“incurring the additional overhead of disk I/O and increased garbage collection”
↩︎ In-memory processing and lazy planning“if any partition of an RDD is lost, it will automatically be recomputed using the transformations that originally created it”
↩︎ Fault tolerance through lineage recomputation“RDDs automatically recover from node failures.”
↩︎ Fault tolerance through lineage recomputation“the replicated ones let you continue running tasks on the RDD without waiting to recompute a lost partition”
↩︎ Fault tolerance through lineage recomputation“Same as the levels above, but replicate each partition on two cluster nodes.”
↩︎ Fault tolerance through lineage recomputation“The Shuffle is an expensive operation since it involves disk I/O, data serialization, and network I/O.”
↩︎ Challenges built into distributed processing“This means that long-running Spark jobs may consume a large amount of disk space.”
↩︎ Challenges built into distributed processing“if any partition of an RDD is lost, it will automatically be recomputed using the transformations that originally created it”
↩︎ Key concept“All the storage levels provide full fault tolerance by recomputing lost data”
↩︎ Exam trap 1“This allows future actions to be much faster (often by more than 10x).”
↩︎ Prediction“When data does not fit in memory Spark will spill these tables to disk, incurring the additional overhead of disk I/O and increased garbage collection.”
↩︎ Checkpoint - 5.https://spark.apache.org/docs/latest/cluster-overview.htmlSecondary source
“runs tasks and keeps data in memory or disk storage across them”
↩︎ In-memory processing and lazy planning“Each job gets divided into smaller sets of tasks called stages that depend on each other”
↩︎ In-memory processing and lazy planning“Because the driver schedules tasks on the cluster”
↩︎ Fault tolerance through lineage recomputation“data cannot be shared across different Spark applications (instances of SparkContext) without writing it to an external storage system”
↩︎ Challenges built into distributed processing - 6.
“Spill happens when Spark runs low on execution memory”
↩︎ In-memory processing and lazy planning“The other cores sit idle while those tasks finish, which results in poor cluster utilization and longer jobs.”
↩︎ Challenges built into distributed processing“Spill happens when Spark runs low on execution memory”
↩︎ Exam trap 4 - 7.https://docs.databricks.com/aws/en/spark/faqOfficial docs
“Apache Spark uses a lazy execution model”
↩︎ In-memory processing and lazy planning“consider all transformations triggered by a given action at once”
↩︎ In-memory processing and lazy planning“Manually caching data or returning preview results in production pipelines can interrupt these optimizations and lead to increases in cost and latency.”
↩︎ In-memory processing and lazy planning - 8.
“When an executor is removed, Spark reruns its unfinished tasks.”
↩︎ Fault tolerance through lineage recomputation - 9.
“The job continues, but Spark must rerun those tasks, which slows the job and might cause stage or job failures.”
↩︎ Fault tolerance through lineage recomputation“Databricks recommends using an on-demand instance for the driver node so that the cloud provider can't reclaim it.”
↩︎ Challenges built into distributed processing“If the cloud provider reclaims the driver, the job fails immediately.”
↩︎ Exam trap 2 - 10.
“When executors run out of memory, Spark removes them and retries their tasks.”
↩︎ Challenges built into distributed processing - 11.
“Small input might land in a single partition, so one task runs the whole UDF.”
↩︎ When Spark is the wrong tool“Adding workers doesn't help, because Spark can't spread the work across them.”
↩︎ Prediction - 12.https://docs.databricks.com/aws/en/optimizations/aqeOfficial docs
“Very small tasks have worse I/O throughput and tend to suffer more from scheduling overhead and task setup overhead.”
↩︎ When Spark is the wrong tool“Very small tasks have worse I/O throughput and tend to suffer more from scheduling overhead and task setup overhead.”
↩︎ Exam trap 3“create a few oversized tasks that hold up the whole stage”
↩︎ Checkpoint