CertSafari
    Databricks Certified Associate Developer for Apache Spark· Lessons

    Domain 4 · Lesson 22/32

    Spark partition tuning: repartition, coalesce, shuffles and data skew

    Implement performance tuning strategies & optimize cluster utilization, including partitioning, repartitioning, coalescing, identifying data skew, and reducing shuffling

    17 min read
    3.33% of exam
    12 sources
    Published 21 Oct 2026
    Docs as of 17 Oct 2026

    What you will be able to do

    • Tell when a stage has too few or too many partitions, and pick the setting or call that changes the count
    • Choose between repartition(), repartitionByRange(), coalesce() and the matching SQL partitioning hints, given a target partition count and a cost budget
    • Cut shuffles by filtering early, broadcasting small join sides and dropping redundant repartitions, then check the result for Exchange nodes in explain()
    • Spot data skew from a stage's task durations and fix it with AQE, salting or the REBALANCE hint

    Key concept

    Idle cores behind a few busy tasks — A stage finishes only when its slowest task finishes. Every tuning lever on this page, from partition count and coalescing to shuffle reduction and skew fixes, aims to give all cores evenly sized work so none of them sit idle waiting for a straggler.

    1.Partition count as the utilization lever

    The number of partitions decides how much of the cluster a stage can use, because each partition becomes one task. Where the initial count comes from (file splits, spark.sql.shuffle.partitions) is covered in Partitions and where they come from. This page treats that count as something you tune, and it goes wrong in two opposite directions.

    Too few partitions means large partitions and too little parallelism. If a stage has 4 partitions on a 32-core cluster, 28 cores do nothing. Each oversized task may also run out of execution memory, which Spark uses for shuffles, joins, sorts and aggregations, and then spill data to disk. Spill slows the task down further.

    Too many partitions means tiny tasks whose fixed costs outweigh their work. Very small tasks get worse I/O throughput and pay more in scheduling and task setup overhead. Merging them into fewer, reasonably sized tasks saves resources and raises cluster throughput.

    spark.sql.shuffle.partitions (default 200) sets the partition count only when data is shuffled for joins or aggregations. A job that reads a table, filters it and writes it has no shuffle, so changing this setting right before the write leaves the number of output partitions, and so the number of output files, where it was. To control output files, change the partitioning of the DataFrame itself, as the next section shows.

    Repartitioning is the direct way to set the count on a DataFrame. df.repartition(10) returns a DataFrame with 10 partitions whatever the current count, so it is the call for a stage that has too few partitions for the cores available. Pass columns as well, as in df.repartition(7, "age"), and the rows are hash partitioned on those columns. If you pass no number, Spark uses the default partition count. Because repartitioning moves rows across the network, reserve it for cases where the new layout is worth a shuffle.

    Hand-tuning the shuffle partition count for every query rarely pays off, because the right number depends on data size, and data size changes from stage to stage. A better default is to rely on AQE coalescing of shuffle partitions. Start with a generous initial count and let Spark merge small post-shuffle partitions at runtime, based on the actual size of the data.

    Checkpoint 1 of 9· Check yourself

    A job reads a Parquet table, filters it and writes the result. It produces thousands of tiny output files. An engineer sets spark.sql.shuffle.partitions to 50 just before the write and runs the job again. What happens to the number of output files?

    Sources12345

    2.repartition() vs coalesce(), by column, by range, and as SQL hints

    repartition() always shuffles and can go to any partition count, up or down. coalesce() avoids a full shuffle but can only lower the count. That one distinction decides most choices between them.

    repartition(n) redistributes every row across the network into n partitions. Passing columns, as in repartition(7, "age"), makes the result hash partitioned on those columns, so rows with the same value end up together. That has a catch. If every row has the same value in the partitioning column (say, after filtering to region == "EMEA"), then repartition(8, "region") does not give you eight even partitions. All the rows hash to the same place. repartitionByRange(n, col) also shuffles, but it places rows into sorted ranges of the column. It needs at least one column, and because it samples the data to estimate the ranges, its output can vary between runs.

    coalesce(n) creates a narrow dependency: going from 1000 partitions to 100, each new partition takes over 10 existing ones and no rows cross the network. That makes it the cheap way to cut the file count before a write. Take a filtered DataFrame that still has 1000 mostly empty partitions and needs to land as exactly 8 files: coalesce(8) merges them in place, while repartition(8) pays for a full shuffle to reach the same count. Calling persist() instead would not help either, because caching stores data in memory and does not change the partition count.

    The cost of coalesce shows up when it is drastic. coalesce(1) can push the upstream computation onto a single node, because the narrow dependency means the work that produces those partitions runs inside the few remaining tasks. If the upstream work is heavy, repartition(1) can be faster even though it adds a shuffle. The shuffle lets the upstream partitions run in parallel first, and only the final merge is serial.

    coalesce(1) collapses a 3-partition DataFrame into a single partition without a shufflepython
    from pyspark.sql import functions as sf
    spark.range(0, 10, 1, 3).coalesce(1).select(
        sf.spark_partition_id().alias("partition")
    ).distinct().sort("partition").show()
    # +---------+
    # |partition|
    # +---------+
    # |        0|
    # +---------+

    SQL users get the same controls through partitioning hints. COALESCE, REPARTITION and REPARTITION_BY_RANGE are equivalent to coalesce, repartition and repartitionByRange, and they are a common way to control the number of files a query writes. COALESCE takes only a partition number. REPARTITION takes a number, columns, both or neither. REPARTITION_BY_RANGE requires columns. A fourth hint, REBALANCE, evens out partition sizes for writes and returns in the skew section.

    Partitioning hints in SQL, mirroring the DataFrame methodssql
    SELECT /*+ COALESCE(3) */ * FROM t;
    SELECT /*+ REPARTITION(3) */ * FROM t;
    SELECT /*+ REPARTITION(c) */ * FROM t;
    SELECT /*+ REPARTITION(3, c) */ * FROM t;
    SELECT /*+ REPARTITION */ * FROM t;
    SELECT /*+ REPARTITION_BY_RANGE(c) */ * FROM t;
    SELECT /*+ REPARTITION_BY_RANGE(3, c) */ * FROM t;
    Partitioning calls and hints compared
    DataFrame callSQL hintWhat it doesUse when
    repartition(n) / repartition(n, cols)REPARTITIONFull shuffle to any count; hash partitioned when columns are givenRaising the count, or grouping rows by key before later work
    repartitionByRange(n, cols)REPARTITION_BY_RANGEFull shuffle into sorted ranges; needs a columnOutput should be split into value ranges of a column
    coalesce(n)COALESCEMerges existing partitions, no shuffle; can only reduceCutting the file count before a write, without going drastically low
    (none)REBALANCEEvens out output partition sizes and splits skewed ones; ignored without AQEWriting a table without files that are too small or too big

    Checkpoint 2 of 9· Fill the gap

    This sample collapses a 3-partition DataFrame into one partition without a shuffle. Which method fills the blank?

    from pyspark.sql import functions as sf
    spark.range(0, 10, 1, 3). ? (1).select(
        sf.spark_partition_id().alias("partition")
    ).distinct().sort("partition").show()
    # +---------+
    # |partition|
    # +---------+
    # |        0|
    # +---------+

    Checkpoint 3 of 9· Check yourself

    An ETL job runs a heavy transformation across 500 partitions and must write a single file. With coalesce(1) at the end, the whole job crawls through one task. Which change is most likely to speed it up?

    Checkpoint 4 of 9· Exam question

    A DataFrame with 1000 partitions is filtered down to a small result set. An engineer wants the final write to produce exactly 8 output files while spending as little execution time as possible on the operation itself: result = spark.read.table("sales.transactions").filter(col("region") == "EMEA") result.write.mode("overwrite").parquet("/mnt/output/emea_sales") Which change achieves this most efficiently?

    Checkpoint 5 of 9· Exam question

    Which statement correctly distinguishes repartition() from coalesce() on a DataFrame?

    Sources567894

    3.Reducing shuffles and confirming it in explain()

    A shuffle is usually the most expensive step in a query, because it has to move data across the network. The mechanics are in Shuffles and wide transformations. In tuning terms, every shuffle you remove, or make smaller, is network traffic and disk I/O you no longer pay for.

    Four habits do most of the work:

    • Filter and project early. If a join returns far fewer rows than it reads, add filters before the join to cut its input rows, and select only the columns you need. Less data in means less data shuffled.
    • Broadcast the small side of a join. Marking a small DataFrame with broadcast(), or using the BROADCAST hint, copies it to every executor so the large side never has to be shuffled. The mechanism is covered in Broadcast hash join.
    • Do not repartition twice. Every repartition() adds its own shuffle step. A repartition() followed by a groupBy on a different key shuffles the data twice, and only the second shuffle shapes the result.
    • Prefer coalesce for shrinking. Reducing the count before a write can usually be done with no shuffle at all.

    Check the physical plan instead of assuming. Each shuffle shows up in explain() output as an Exchange node. A groupBy aggregation produces Exchange hashpartitioning(...) with the shuffle partition count, and a bare REPARTITION(100) hint produces Exchange RoundRobinPartitioning(100). Count the Exchange nodes before and after a change. For a join you meant to broadcast, look for a BroadcastHashJoin node where a SortMergeJoin used to be. If you see a different join node, the plan did not change the way you intended.

    The Exchange line in EXPLAIN output for a GROUP BY: one shuffle into 200 hash partitionstext
    +- Exchange hashpartitioning(k#33, 200), true, [id=#59]

    Checkpoint 6 of 9· Check yourself

    explain() on a pipeline shows two Exchange nodes. The first is Exchange RoundRobinPartitioning(400), from an early df.repartition(400). The second is Exchange hashpartitioning(customer_id, 200), from a later groupBy("customer_id").agg(...). Which change cuts shuffle cost without changing the result?

    Sources101112913

    4.Identifying and fixing data skew

    Data skew means one or a few tasks in a stage take far longer than the rest, usually because a few key values hold most of the rows. In a shuffle, all rows for one key go to one partition, so a single hot customer_id becomes a single huge task while the other cores wait. In the Application → job → stage → task hierarchy, skew shows up at the bottom: one stage whose tasks have very uneven durations.

    To detect skew, open the stage page for the longest stage and look at Summary Metrics. Compare the Max task duration with the 75th percentile. In a healthy stage they are about the same. If Max is 50% or more above the 75th percentile, suspect skew, and also compare input sizes per task, since the slow task is usually the one that read far more data. Spill is a separate signal that appears as spill statistics in the stage details at the top of the page. A single straggler is not a memory or GC problem, so adding memory to every executor still leaves one task doing most of the work. Small source files and broadcast timeouts don't explain it either. Compacting files with OPTIMIZE helps scans, not a hot key, and spark.sql.broadcastTimeout only applies to broadcast joins, not to a groupBy.

    To fix skew, try the remedies in this order:

    1. Check whether AQE already handled it. For a sort merge join or shuffle hash join, AQE skew join handling splits oversized partitions automatically. It only acts when a partition exceeds both spark.sql.adaptive.skewJoin.skewedPartitionFactor times the median and spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes, so lower those if your skewed partitions fall below them. AQE does not touch aggregations or broadcast joins, which are never skew-optimized.
    2. Salt the hot keys. Add a random bucket suffix to the skewed key, aggregate on the salted key, then aggregate the partial results again on the original key. The hot key's rows now spread across many tasks in the first stage, and the second stage combines only a handful of partial results.
    3. Broadcast instead of shuffling. If the skewed key sits on the large side of a join with a small table, broadcasting the small table removes the shuffle that concentrated the key.
    4. Rebalance skewed writes. If the skewed stage writes a table, the REBALANCE hint splits oversized output partitions. Spark ignores it unless AQE is enabled.

    Checkpoint 7 of 9· Check yourself

    On the stage page for a slow groupBy stage, Summary Metrics show a 75th percentile task duration of 40 seconds and a Max of 22 minutes. The slowest task also read far more input than the others. Which diagnosis and next step fit?

    Checkpoint 8 of 9· Match them up

    Match each skew remedy to the situation it fits best

    Tap a term, then the definition that fits it.

    Checkpoint 9 of 9· Exam question

    In the Spark UI, a job's shuffle stage shows 199 tasks finishing in under 10 seconds while a single task runs for 25 minutes and dominates the stage's wall-clock time. The stage performs a groupBy("customer_id").agg(sum("amount")) where one customer_id accounts for a large share of all rows. What is this symptom, and which change directly addresses its root cause?

    Sources21104

    Sources

    (12)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Very small tasks have worse I/O throughput and tend to suffer more from scheduling overhead and task setup overhead.”
      ↩︎ Partition count as the utilization lever
      “The default number of partitions to use when shuffling data for joins or aggregations.”
      ↩︎ Partition count as the utilization lever
      “The current or final plan has a different physical join node than the initial plan.”
      ↩︎ Reducing shuffles and confirming it in explain()
      “AQE treats a partition as skewed only when both of these conditions are true:”
      ↩︎ Identifying and fixing data skew
      “The default number of partitions to use when shuffling data for joins or aggregations.”
      ↩︎ Exam trap 1
    2. 2.
      “Spill happens when Spark runs low on execution memory, which Spark uses for shuffles, joins, sorts, and aggregations.”
      ↩︎ Partition count as the utilization lever
      “If the Max duration exceeds the 75th percentile by 50% or more, the stage might have skew.”
      ↩︎ Identifying and fixing data skew
      “The stage details at the top of the stage page show spill statistics when the stage has spill”
      ↩︎ Identifying and fixing data skew
      “Broadcast joins are never skew-optimized”
      ↩︎ Identifying and fixing data skew
      “append a random bucket suffix to the skewed keys, and then group and aggregate in two stages”
      ↩︎ Identifying and fixing data skew
      “Spark ignores the hint if AQE isn't enabled.”
      ↩︎ Identifying and fixing data skew
      “The other cores sit idle while those tasks finish, which results in poor cluster utilization and longer jobs.”
      ↩︎ Key concept
      “Skew happens when one or a few tasks take much longer than the rest.”
      ↩︎ Exam trap 4
      “If a few key values hold most of the data in a join or aggregation, append a random bucket suffix to the skewed keys”
      ↩︎ Checkpoint
    3. 3.
      “If there are too few partitions, then the data size of each partition may be very large”
      ↩︎ Partition count as the utilization lever
      “Shuffle is an expensive operator as it needs to move data across the network”
      ↩︎ Reducing shuffles and confirming it in explain()
    4. 4.
      “You do not need to set a proper shuffle partition number to fit your dataset.”
      ↩︎ Partition count as the utilization lever
      “they can be used for performance tuning and reducing the number of output files”
      ↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints
      “Spark SQL can cache tables using an in-memory columnar format”
      ↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints
      “Timeout in seconds for the broadcast wait time in broadcast joins”
      ↩︎ Identifying and fixing data skew
      “Configures the number of partitions to use when shuffling data for joins or aggregations.”
      ↩︎ Checkpoint
    5. 5.
      “can be an int to specify the target number of partitions or a Column.”
      ↩︎ Partition count as the utilization lever
      “Returns a new DataFrame partitioned by the given partitioning expressions. The resulting DataFrame is hash partitioned.”
      ↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints
    6. 6.
      “this operation results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle”
      ↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints
      “this may result in your computation taking place on fewer nodes than you like”
      ↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints
      “This will add a shuffle step, but means the current upstream partitions will be executed in parallel”
      ↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints
      “If a larger number of partitions is requested, it will stay at the current number of partitions.”
      ↩︎ Exam trap 2
      “If a larger number of partitions is requested, it will stay at the current number of partitions.”
      ↩︎ Prediction
      “To avoid this, you can call repartition(). This will add a shuffle step, but means the current upstream partitions will be executed in parallel”
      ↩︎ Checkpoint
      “This will add a shuffle step, but means the current upstream partitions will be executed in parallel”
      ↩︎ Checkpoint
    7. 9.
      “COALESCE, REPARTITION, and REPARTITION_BY_RANGE hints are supported and are equivalent to coalesce, repartition, and repartitionByRange Dataset APIs, respectively.”
      ↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints
      “Exchange RoundRobinPartitioning(100), false, [id=#121]”
      ↩︎ Reducing shuffles and confirming it in explain()
    8. 10.
      “add filters before the join to reduce input rows”
      ↩︎ Reducing shuffles and confirming it in explain()
      “It chose a broadcast join instead of shuffling data across the cluster”
      ↩︎ Reducing shuffles and confirming it in explain()
      “Run OPTIMIZE to compact small files.”
      ↩︎ Identifying and fixing data skew

    Spotted a mistake, or was something unclear? Tell us.