CertSafari
    Databricks Certified Associate Developer for Apache Spark· Lessons

    Domain 1 · Lesson 1/32

    Spark advantages and challenges: unified engine, memory, lineage, limits

    Identify the advantages and challenges of implementing Spark.

    16 min read
    2.86% of exam
    12 sources
    Published 21 Oct 2026
    Docs as of 17 Oct 2026

    What you will be able to do

    • Explain why one engine for batch, SQL, streaming and ML is Spark's headline advantage when consolidating separate systems
    • Explain why keeping intermediate data in memory speeds up iterative and multi-step jobs
    • Describe how Spark recovers lost partitions through lineage, and how this differs from replication
    • Name the built-in costs of running Spark: shuffles, spill and skew, the single driver, and isolated applications
    • Recognise workloads where Spark's distribution overhead brings no benefit

    Key concept

    Lineage recomputation — Spark records the chain of transformations that produced each partition. When a partition is lost, Spark replays that chain to rebuild it, so it can tolerate failures without keeping a second copy of the data.

    1.One engine for batch, SQL, streaming and machine learning

    Spark's headline advantage is that one engine runs many kinds of work. A company that runs nightly ETL on one system, analyst SQL on a second and model training on a third has to move data between them and keep three platforms working. Spark replaces all three with a single stack: Spark SQL and DataFrames for structured processing, Structured Streaming for continuous data, and MLlib for machine learning. All of them run on the same cluster and the same execution engine.

    The unification goes deeper than a shared installation. Spark SQL and the DataFrame and Dataset APIs all compile down to the same engine, whichever API or language expresses the computation. A developer can therefore switch between SQL and DataFrame code inside one pipeline without changing how it runs. Batch and streaming share the engine too: Spark and Structured Streaming have a unified architecture for both, and streaming can run either triggered or continuously, so each workload can trade cost against latency.

    Machine learning belongs in the same picture. MLlib is optimised for large-scale machine learning algorithms and statistical modelling. Spark is especially strong at the data-heavy work around models: ETL, analytics reporting, feature engineering and data preprocessing. A demand-forecasting job can therefore read the tables that the nightly ETL just wrote, on the same cluster, with no hand-off to another system.

    Checkpoint 1 of 8· Check yourself

    A retailer runs nightly ETL on one system, analyst SQL on a second and demand-forecast model training on a third. Data has to be copied between them, and the team maintains three platforms. Which Spark advantage solves this problem directly?

    Checkpoint 2 of 8· Exam question

    A retail company runs separate Hadoop MapReduce jobs for nightly ETL, a dedicated SQL engine for analyst reporting, and a standalone cluster for demand-forecasting models, and wants to consolidate onto one platform without losing performance. Which advantage of Apache Spark most directly supports this consolidation?

    Sources1234

    2.In-memory processing and lazy planning

    Spark keeps intermediate data in memory between steps instead of writing it out and reading it back. An executor is a process that runs tasks and keeps data in memory or on disk across them for the whole application. When a dataset is persisted, each node keeps the partitions it computed in memory and reuses them for later actions on that dataset or on datasets derived from it. Reuse is where the speedup comes from, often more than 10x. An iterative algorithm that passes over the same data twenty times reads it from storage once, not twenty times. That is why caching is described as a key tool for iterative algorithms and fast interactive use.

    In-memory processing does not remove the ordering of work. Each job is split into stages that depend on each other, and a later stage still waits for the stage that produces its input. Memory is also finite. Spill happens when Spark runs low on execution memory, which it uses for shuffles, joins, sorts and aggregations. The data that does not fit is written to disk, adding disk I/O and extra garbage collection. Spark runs on the JVM, and its garbage collector is a cost to manage, not something Spark has engineered away.

    Lazy evaluation adds a second advantage on top of memory. Transformations only add logic to a plan, and nothing runs until an action asks for a result (the mechanics are in Transformations, actions and lazy evaluation). An engine that runs each step as soon as it is written has to optimise one step at a time. Spark sees the whole plan first, so its optimisations can consider every transformation behind an action together and choose a plan based on how the data is physically laid out.

    Checkpoint 3 of 8· Check yourself

    A cluster was sized for in-memory processing. During a large join and aggregation, its executors still show heavy disk writes and longer garbage-collection pauses. What is the most likely explanation?

    Checkpoint 4 of 8· Exam question

    An engineer migrates a nightly aggregation pipeline from disk-based MapReduce to Spark and observes a large runtime improvement on a job that repeatedly scans and re-scans the same intermediate dataset across several stages. Which characteristic of Spark's design explains this speedup?

    Sources5467

    3.Fault tolerance through lineage recomputation

    When an executor is lost, Spark rebuilds the data rather than restoring a copy. Every partition has a recorded lineage, the transformations that produced it from its input. If a partition disappears with its executor, Spark replays those transformations on the original data and produces the partition again. This is why RDDs automatically recover from node failures, and why cached data is fault-tolerant without being stored twice.

    Recovery has two parts: rebuilding the data and rescheduling the work. The driver schedules every task on the cluster. When an executor is removed, Spark reruns its unfinished tasks on the executors that remain. Consider a worker on a spot instance that the cloud provider reclaims. Its executors terminate and their active tasks fail. The job continues, and Spark reruns those tasks elsewhere. The result is still correct, but the job takes longer, and frequent losses can make a stage or the whole job fail.

    What happens when a node is lost during a job
    EventWhat Spark doesOutcome
    A worker spot instance is reclaimedIts executors terminate, active tasks fail, and Spark reruns those tasksThe job continues but runs slower
    Workers are lost repeatedlySpark keeps rerunning the failed tasksStage or job failures become possible
    The driver instance is reclaimedThere is no rerun, because the driver is the process that schedules tasksThe job fails immediately

    Replication is available, but only if you ask for it. Storage levels ending in _2, such as MEMORY_ONLY_2 and MEMORY_AND_DISK_2, keep each partition on two cluster nodes. With the default MEMORY_ONLY, a lost partition is recomputed. All storage levels give full fault tolerance through recomputation. The replicated levels only save the time spent waiting for a lost partition to be rebuilt, which matters for a low-latency case such as Spark serving requests from a web application.

    Recovery with and without a replicated storage level
    Storage levelCopies of each partitionWhen a partition is lost
    MEMORY_ONLY (default for cache())OneRecomputed from its lineage
    MEMORY_ONLY_2, MEMORY_AND_DISK_2Two cluster nodesTasks continue on the replica without waiting for recomputation

    Checkpoint 5 of 8· Match them up

    A long-running Spark job hits each of these failures. Match each failure to what Spark does next.

    Tap a term, then the definition that fits it.

    Checkpoint 6 of 8· Exam question

    During a long-running Spark job, one executor is terminated after the underlying VM is reclaimed by the cloud provider. The job continues and eventually finishes with correct results. Which two statements correctly describe how Spark achieves this fault tolerance? (Choose 2 answers)(Select 2)

    0 of 2 selected

    Sources4895

    4.Challenges built into distributed processing

    The shuffle is the main performance cost of distributed processing. Operations such as groupByKey, reduceByKey, join and repartition need all values for a key in one place, so Spark copies data across executors and machines. That involves disk I/O, data serialization and network I/O, and it also leaves intermediate files on disk. In a long-running application these files can take up a large amount of disk space. Shuffles cannot be avoided whenever data has to be regrouped, and much of Spark tuning comes down to shuffling less (Shuffles and wide transformations).

    Memory pressure is the second built-in challenge. Shuffle operations build in-memory structures on the heap. When the data does not fit, Spark spills to disk and the stage slows down. Skew makes this worse. When one or a few tasks take much longer than the rest, the other cores sit idle until they finish, so a 200-core cluster can spend most of a stage waiting on one oversized partition. When executors run out of memory, Spark removes them and retries their tasks, and repeated failures make the stage and the job fail. Sizing memory per core is a real engineering task, covered in Executor and driver memory, spill and cluster sizing.

    The design also concentrates risk in one process. Each application has a single driver that schedules its tasks, and executors connect back to it throughout the run (the roles are covered in Driver, worker nodes, executors and the cluster manager). Losing a worker costs time. Losing the driver ends the job. For this reason the driver should run on an on-demand instance even when the workers use spot capacity, and it should sit close to the workers on the network.

    Isolation between applications brings a cost of its own. Each application gets its own executors, and tasks from different applications run in different JVMs. This keeps one team's job from disturbing another's. It also means two Spark applications cannot share an in-memory dataset. One must write the data to external storage before the other can read it.

    Checkpoint 7 of 8· Check yourself

    On a 200-core cluster, 199 tasks of a stage finish in two minutes. The last task, which reads one very large partition, runs for forty minutes. The team proposes doubling the number of workers. How much would that shorten this stage?

    Sources461095

    5.When Spark is the wrong tool

    Spark pays off when the same operation runs over a large dataset. Its strength is data parallelism: split the data into partitions and apply the same logic to each partition at once. It is the recommended choice for large-scale joins, filters and aggregations. Everything else Spark does, including driver planning, task scheduling, shuffles and executors, is overhead that only a large dataset pays back.

    On small data the overhead becomes the workload. Small input may fit in a single partition, so one task does all the work and extra workers do nothing. Splitting the data into many tiny tasks does not help either. Very small tasks have worse I/O throughput and suffer more from scheduling and task setup overhead. A workload that reads or changes only a few records falls into the same trap. A request for one record still becomes a job planned by the driver and run as tasks on executors, and with nothing to split across partitions, that machinery adds cost and buys no parallelism.

    Some heavy workloads are the wrong shape for Spark. For task parallelism, many independent tasks running at once, Ray is the better fit. Reinforcement learning, simulation modelling, hyperparameter search and deep learning training are listed as workloads where Spark is less optimised. One common pattern uses Spark for data retrieval and preparation, then hands the compute-heavy stages to Ray running in the same environment.

    Matching the workload to the engine
    WorkloadBetter fitReason
    ETL, analytics reporting, feature engineering on large tablesSparkData parallelism: the same operation applied to each element of a large dataset
    Large-scale ML algorithms and statistical modellingSpark (MLlib)Optimised for large-scale machine learning
    Hyperparameter search, simulation, deep learning trainingRayTask parallelism: independent tasks run concurrently

    Checkpoint 8 of 8· Check yourself

    Which plan fits a pipeline that builds features from large tables and then runs a hyperparameter search that trains hundreds of independent model variants?

    Sources31112

    Sources

    (12)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “has a unified architecture for batch and streaming processing”
      ↩︎ One engine for batch, SQL, streaming and machine learning
      “Streaming processing can be run in both triggered and continuous manners”
      ↩︎ One engine for batch, SQL, streaming and machine learning
    2. 2.
      “the same execution engine is used, independent of which API/language you are using to express the computation”
      ↩︎ One engine for batch, SQL, streaming and machine learning
      “When computing a result, the same execution engine is used, independent of which API/language you are using to express the computation.”
      ↩︎ Checkpoint
    3. 3.
      “Spark's MLlib and SparkML libraries are optimized for large-scale machine learning algorithms and statistical modeling.”
      ↩︎ One engine for batch, SQL, streaming and machine learning
      “It's ideal for ETL, analytics reporting, feature engineering, and data preprocessing.”
      ↩︎ One engine for batch, SQL, streaming and machine learning
      “Spark excels at data parallelism - Apply the same operation to each element of a large dataset.”
      ↩︎ When Spark is the wrong tool
      “Use Ray for workloads where Spark is less optimized”
      ↩︎ When Spark is the wrong tool
      “Use Spark for data-intensive tasks and switch to Ray for stages that require heavy computation.”
      ↩︎ When Spark is the wrong tool
    4. 4.
      “This nomenclature comes from MapReduce and does not directly relate to Spark’s map and reduce operations.”
      ↩︎ One engine for batch, SQL, streaming and machine learning
      “Caching is a key tool for iterative algorithms and fast interactive use.”
      ↩︎ In-memory processing and lazy planning
      “incurring the additional overhead of disk I/O and increased garbage collection”
      ↩︎ In-memory processing and lazy planning
      “if any partition of an RDD is lost, it will automatically be recomputed using the transformations that originally created it”
      ↩︎ Fault tolerance through lineage recomputation
      “RDDs automatically recover from node failures.”
      ↩︎ Fault tolerance through lineage recomputation
      “the replicated ones let you continue running tasks on the RDD without waiting to recompute a lost partition”
      ↩︎ Fault tolerance through lineage recomputation
      “Same as the levels above, but replicate each partition on two cluster nodes.”
      ↩︎ Fault tolerance through lineage recomputation
      “The Shuffle is an expensive operation since it involves disk I/O, data serialization, and network I/O.”
      ↩︎ Challenges built into distributed processing
      “This means that long-running Spark jobs may consume a large amount of disk space.”
      ↩︎ Challenges built into distributed processing
      “if any partition of an RDD is lost, it will automatically be recomputed using the transformations that originally created it”
      ↩︎ Key concept
      “All the storage levels provide full fault tolerance by recomputing lost data”
      ↩︎ Exam trap 1
      “This allows future actions to be much faster (often by more than 10x).”
      ↩︎ Prediction
      “When data does not fit in memory Spark will spill these tables to disk, incurring the additional overhead of disk I/O and increased garbage collection.”
      ↩︎ Checkpoint
    5. 5.
      “runs tasks and keeps data in memory or disk storage across them”
      ↩︎ In-memory processing and lazy planning
      “Each job gets divided into smaller sets of tasks called stages that depend on each other”
      ↩︎ In-memory processing and lazy planning
      “Because the driver schedules tasks on the cluster”
      ↩︎ Fault tolerance through lineage recomputation
      “data cannot be shared across different Spark applications (instances of SparkContext) without writing it to an external storage system”
      ↩︎ Challenges built into distributed processing
    6. 6.
      “Spill happens when Spark runs low on execution memory”
      ↩︎ In-memory processing and lazy planning
      “The other cores sit idle while those tasks finish, which results in poor cluster utilization and longer jobs.”
      ↩︎ Challenges built into distributed processing
      “Spill happens when Spark runs low on execution memory”
      ↩︎ Exam trap 4
    7. 7.
      “Apache Spark uses a lazy execution model”
      ↩︎ In-memory processing and lazy planning
      “consider all transformations triggered by a given action at once”
      ↩︎ In-memory processing and lazy planning
      “Manually caching data or returning preview results in production pipelines can interrupt these optimizations and lead to increases in cost and latency.”
      ↩︎ In-memory processing and lazy planning
    8. 9.
      “The job continues, but Spark must rerun those tasks, which slows the job and might cause stage or job failures.”
      ↩︎ Fault tolerance through lineage recomputation
      “Databricks recommends using an on-demand instance for the driver node so that the cloud provider can't reclaim it.”
      ↩︎ Challenges built into distributed processing
      “If the cloud provider reclaims the driver, the job fails immediately.”
      ↩︎ Exam trap 2
    9. 10.
      “When executors run out of memory, Spark removes them and retries their tasks.”
      ↩︎ Challenges built into distributed processing
    10. 11.
      “Small input might land in a single partition, so one task runs the whole UDF.”
      ↩︎ When Spark is the wrong tool
      “Adding workers doesn't help, because Spark can't spread the work across them.”
      ↩︎ Prediction
    11. 12.
      “Very small tasks have worse I/O throughput and tend to suffer more from scheduling overhead and task setup overhead.”
      ↩︎ When Spark is the wrong tool
      “Very small tasks have worse I/O throughput and tend to suffer more from scheduling overhead and task setup overhead.”
      ↩︎ Exam trap 3
      “create a few oversized tasks that hold up the whole stage”
      ↩︎ Checkpoint

    Ready to test yourself?

    Practise the 11 questions on this subdomain.

    Spotted a mistake, or was something unclear? Tell us.