What you will be able to do
- Tell when a stage has too few or too many partitions, and pick the setting or call that changes the count
- Choose between repartition(), repartitionByRange(), coalesce() and the matching SQL partitioning hints, given a target partition count and a cost budget
- Cut shuffles by filtering early, broadcasting small join sides and dropping redundant repartitions, then check the result for Exchange nodes in explain()
- Spot data skew from a stage's task durations and fix it with AQE, salting or the REBALANCE hint
Key concept
Idle cores behind a few busy tasks — A stage finishes only when its slowest task finishes. Every tuning lever on this page, from partition count and coalescing to shuffle reduction and skew fixes, aims to give all cores evenly sized work so none of them sit idle waiting for a straggler.
1.Partition count as the utilization lever
The number of partitions decides how much of the cluster a stage can use, because each partition becomes one task. Where the initial count comes from (file splits, spark.sql.shuffle.partitions) is covered in Partitions and where they come from. This page treats that count as something you tune, and it goes wrong in two opposite directions.
Too few partitions means large partitions and too little parallelism. If a stage has 4 partitions on a 32-core cluster, 28 cores do nothing. Each oversized task may also run out of execution memory, which Spark uses for shuffles, joins, sorts and aggregations, and then spill data to disk. Spill slows the task down further.
Too many partitions means tiny tasks whose fixed costs outweigh their work. Very small tasks get worse I/O throughput and pay more in scheduling and task setup overhead. Merging them into fewer, reasonably sized tasks saves resources and raises cluster throughput.
spark.sql.shuffle.partitions (default 200) sets the partition count only when data is shuffled for joins or aggregations. A job that reads a table, filters it and writes it has no shuffle, so changing this setting right before the write leaves the number of output partitions, and so the number of output files, where it was. To control output files, change the partitioning of the DataFrame itself, as the next section shows.
Repartitioning is the direct way to set the count on a DataFrame. df.repartition(10) returns a DataFrame with 10 partitions whatever the current count, so it is the call for a stage that has too few partitions for the cores available. Pass columns as well, as in df.repartition(7, "age"), and the rows are hash partitioned on those columns. If you pass no number, Spark uses the default partition count. Because repartitioning moves rows across the network, reserve it for cases where the new layout is worth a shuffle.
Hand-tuning the shuffle partition count for every query rarely pays off, because the right number depends on data size, and data size changes from stage to stage. A better default is to rely on AQE coalescing of shuffle partitions. Start with a generous initial count and let Spark merge small post-shuffle partitions at runtime, based on the actual size of the data.
Checkpoint 1 of 9· Check yourself
A job reads a Parquet table, filters it and writes the result. It produces thousands of tiny output files. An engineer sets spark.sql.shuffle.partitions to 50 just before the write and runs the job again. What happens to the number of output files?
spark.sql.shuffle.partitions sets the partition count only when data is shuffled for joins or aggregations. A read-filter-write job has no shuffle, so the output partitioning stays the same. To control the number of output files, change the DataFrame's own partitioning with coalesce() or repartition().
“Configures the number of partitions to use when shuffling data for joins or aggregations.”Source: spark.apache.org
2.repartition() vs coalesce(), by column, by range, and as SQL hints
repartition() always shuffles and can go to any partition count, up or down. coalesce() avoids a full shuffle but can only lower the count. That one distinction decides most choices between them.
repartition(n) redistributes every row across the network into n partitions. Passing columns, as in repartition(7, "age"), makes the result hash partitioned on those columns, so rows with the same value end up together. That has a catch. If every row has the same value in the partitioning column (say, after filtering to region == "EMEA"), then repartition(8, "region") does not give you eight even partitions. All the rows hash to the same place. repartitionByRange(n, col) also shuffles, but it places rows into sorted ranges of the column. It needs at least one column, and because it samples the data to estimate the ranges, its output can vary between runs.
coalesce(n) creates a narrow dependency: going from 1000 partitions to 100, each new partition takes over 10 existing ones and no rows cross the network. That makes it the cheap way to cut the file count before a write. Take a filtered DataFrame that still has 1000 mostly empty partitions and needs to land as exactly 8 files: coalesce(8) merges them in place, while repartition(8) pays for a full shuffle to reach the same count. Calling persist() instead would not help either, because caching stores data in memory and does not change the partition count.
The cost of coalesce shows up when it is drastic. coalesce(1) can push the upstream computation onto a single node, because the narrow dependency means the work that produces those partitions runs inside the few remaining tasks. If the upstream work is heavy, repartition(1) can be faster even though it adds a shuffle. The shuffle lets the upstream partitions run in parallel first, and only the final merge is serial.
from pyspark.sql import functions as sf
spark.range(0, 10, 1, 3).coalesce(1).select(
sf.spark_partition_id().alias("partition")
).distinct().sort("partition").show()
# +---------+
# |partition|
# +---------+
# | 0|
# +---------+SQL users get the same controls through partitioning hints. COALESCE, REPARTITION and REPARTITION_BY_RANGE are equivalent to coalesce, repartition and repartitionByRange, and they are a common way to control the number of files a query writes. COALESCE takes only a partition number. REPARTITION takes a number, columns, both or neither. REPARTITION_BY_RANGE requires columns. A fourth hint, REBALANCE, evens out partition sizes for writes and returns in the skew section.
SELECT /*+ COALESCE(3) */ * FROM t;
SELECT /*+ REPARTITION(3) */ * FROM t;
SELECT /*+ REPARTITION(c) */ * FROM t;
SELECT /*+ REPARTITION(3, c) */ * FROM t;
SELECT /*+ REPARTITION */ * FROM t;
SELECT /*+ REPARTITION_BY_RANGE(c) */ * FROM t;
SELECT /*+ REPARTITION_BY_RANGE(3, c) */ * FROM t;| DataFrame call | SQL hint | What it does | Use when |
|---|---|---|---|
| repartition(n) / repartition(n, cols) | REPARTITION | Full shuffle to any count; hash partitioned when columns are given | Raising the count, or grouping rows by key before later work |
| repartitionByRange(n, cols) | REPARTITION_BY_RANGE | Full shuffle into sorted ranges; needs a column | Output should be split into value ranges of a column |
| coalesce(n) | COALESCE | Merges existing partitions, no shuffle; can only reduce | Cutting the file count before a write, without going drastically low |
| (none) | REBALANCE | Evens out output partition sizes and splits skewed ones; ignored without AQE | Writing a table without files that are too small or too big |
Checkpoint 2 of 9· Fill the gap
This sample collapses a 3-partition DataFrame into one partition without a shuffle. Which method fills the blank?
from pyspark.sql import functions as sf
spark.range(0, 10, 1, 3). ? (1).select(
sf.spark_partition_id().alias("partition")
).distinct().sort("partition").show()
# +---------+
# |partition|
# +---------+
# | 0|
# +---------+coalesce() creates a narrow dependency that merges existing partitions without a shuffle. repartition(1) would reach one partition too, but only through a full shuffle. repartitionByRange needs a column, and persist() does not change the partition count.
Source: docs.databricks.comCheckpoint 3 of 9· Check yourself
An ETL job runs a heavy transformation across 500 partitions and must write a single file. With coalesce(1) at the end, the whole job crawls through one task. Which change is most likely to speed it up?
A drastic coalesce keeps the narrow dependency, so the upstream work runs inside the single remaining task. repartition(1) adds a shuffle, but the 500 upstream partitions run in parallel first and only the final merge is serial.
“To avoid this, you can call repartition(). This will add a shuffle step, but means the current upstream partitions will be executed in parallel”Source: docs.databricks.com
Checkpoint 4 of 9· Exam question
A DataFrame with 1000 partitions is filtered down to a small result set. An engineer wants the final write to produce exactly 8 output files while spending as little execution time as possible on the operation itself:
result = spark.read.table("sales.transactions").filter(col("region") == "EMEA")
result.write.mode("overwrite").parquet("/mnt/output/emea_sales")
Which change achieves this most efficiently?
Correct answer: A — Insert result = result.coalesce(8) before the write, since it merges existing partitions on the same executors and avoids a full shuffle across the cluster.
- A.
coalesce()reduces the number of partitions by combining existing ones on the same executors, so it can shrink partition count without a full shuffle when the target is fewer partitions than the current count. This makes it the cheapest way to hit exactly 8 output files here. - B.
repartition()always performs a full shuffle to redistribute rows across the new partition count, regardless of direction, so it is more expensive than needed when only reducing the number of partitions. - C.
spark.sql.shuffle.partitionsonly controls the number of partitions produced by shuffle operations such as joins and aggregations; it has no effect on the partition count of a plain filter-then-write DataFrame operation. - D. Hash partitioning on a single low-cardinality column like
regiondoes not guarantee balanced output; rows with the same region hash to the same partition, so file sizes can still be very uneven, and this still forces a full shuffle. - E. Caching a DataFrame stores its existing partitions in memory or on disk; it does not change the partition count at all, so the write would still emit the original number of files.
Checkpoint 5 of 9· Exam question
Which statement correctly distinguishes repartition() from coalesce() on a DataFrame?
Correct answer: A — repartition() always triggers a full shuffle and can increase or decrease the number of partitions, while coalesce() avoids a full shuffle and can only decrease the partition count.
- A. This matches Spark's actual behavior:
repartition()redistributes all rows across the cluster via a full shuffle and supports moving to either more or fewer partitions, whilecoalesce()merges existing partitions on the same executors to reduce the count without a full shuffle. - B. This reverses the two methods' actual behavior; it is
coalesce()that avoids a shuffle and only reduces partition count, whilerepartition()is the one that shuffles and can move in either direction. - C.
coalesce()specifically avoids a full shuffle in the common case of reducing partitions, and it does not guarantee balanced output sizes since it merges adjacent existing partitions rather than rehashing rows evenly. - D. Both methods operate identically regardless of whether Spark runs in local mode or on a multi-node cluster; neither is restricted to a single deployment mode.
3.Reducing shuffles and confirming it in explain()
A shuffle is usually the most expensive step in a query, because it has to move data across the network. The mechanics are in Shuffles and wide transformations. In tuning terms, every shuffle you remove, or make smaller, is network traffic and disk I/O you no longer pay for.
Four habits do most of the work:
- Filter and project early. If a join returns far fewer rows than it reads, add filters before the join to cut its input rows, and select only the columns you need. Less data in means less data shuffled.
- Broadcast the small side of a join. Marking a small DataFrame with
broadcast(), or using theBROADCASThint, copies it to every executor so the large side never has to be shuffled. The mechanism is covered in Broadcast hash join. - Do not repartition twice. Every
repartition()adds its own shuffle step. Arepartition()followed by agroupByon a different key shuffles the data twice, and only the second shuffle shapes the result. - Prefer coalesce for shrinking. Reducing the count before a write can usually be done with no shuffle at all.
Check the physical plan instead of assuming. Each shuffle shows up in explain() output as an Exchange node. A groupBy aggregation produces Exchange hashpartitioning(...) with the shuffle partition count, and a bare REPARTITION(100) hint produces Exchange RoundRobinPartitioning(100). Count the Exchange nodes before and after a change. For a join you meant to broadcast, look for a BroadcastHashJoin node where a SortMergeJoin used to be. If you see a different join node, the plan did not change the way you intended.
+- Exchange hashpartitioning(k#33, 200), true, [id=#59]Checkpoint 6 of 9· Check yourself
explain() on a pipeline shows two Exchange nodes. The first is Exchange RoundRobinPartitioning(400), from an early df.repartition(400). The second is Exchange hashpartitioning(customer_id, 200), from a later groupBy("customer_id").agg(...). Which change cuts shuffle cost without changing the result?
Every repartition() adds its own shuffle. Here the groupBy shuffles the data again by customer_id, so the first Exchange is wasted work. Removing it leaves one Exchange in the plan. Caching does not change partitioning, and broadcast() is for the small side of a join.
“This will add a shuffle step, but means the current upstream partitions will be executed in parallel”Source: docs.databricks.com
4.Identifying and fixing data skew
Data skew means one or a few tasks in a stage take far longer than the rest, usually because a few key values hold most of the rows. In a shuffle, all rows for one key go to one partition, so a single hot customer_id becomes a single huge task while the other cores wait. In the Application → job → stage → task hierarchy, skew shows up at the bottom: one stage whose tasks have very uneven durations.
To detect skew, open the stage page for the longest stage and look at Summary Metrics. Compare the Max task duration with the 75th percentile. In a healthy stage they are about the same. If Max is 50% or more above the 75th percentile, suspect skew, and also compare input sizes per task, since the slow task is usually the one that read far more data. Spill is a separate signal that appears as spill statistics in the stage details at the top of the page. A single straggler is not a memory or GC problem, so adding memory to every executor still leaves one task doing most of the work. Small source files and broadcast timeouts don't explain it either. Compacting files with OPTIMIZE helps scans, not a hot key, and spark.sql.broadcastTimeout only applies to broadcast joins, not to a groupBy.
To fix skew, try the remedies in this order:
- Check whether AQE already handled it. For a sort merge join or shuffle hash join, AQE skew join handling splits oversized partitions automatically. It only acts when a partition exceeds both
spark.sql.adaptive.skewJoin.skewedPartitionFactortimes the median andspark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes, so lower those if your skewed partitions fall below them. AQE does not touch aggregations or broadcast joins, which are never skew-optimized. - Salt the hot keys. Add a random bucket suffix to the skewed key, aggregate on the salted key, then aggregate the partial results again on the original key. The hot key's rows now spread across many tasks in the first stage, and the second stage combines only a handful of partial results.
- Broadcast instead of shuffling. If the skewed key sits on the large side of a join with a small table, broadcasting the small table removes the shuffle that concentrated the key.
- Rebalance skewed writes. If the skewed stage writes a table, the
REBALANCEhint splits oversized output partitions. Spark ignores it unless AQE is enabled.
After salting, one customer's rows are split across several salted keys, so the first aggregation produces several partial sums for that customer. A second aggregation on the original key combines those partials into the final total. Without it, the result would have one row per salt bucket instead of one row per customer.
Checkpoint 7 of 9· Check yourself
On the stage page for a slow groupBy stage, Summary Metrics show a 75th percentile task duration of 40 seconds and a Max of 22 minutes. The slowest task also read far more input than the others. Which diagnosis and next step fit?
A Max duration far above the 75th percentile, with the slow task reading far more data, is the signature of skew. More memory, file compaction and broadcast timeouts do not move a hot key's rows off a single task. Salting spreads them across many tasks.
“If the Max duration exceeds the 75th percentile by 50% or more, the stage might have skew.”Source: docs.databricks.com
Checkpoint 8 of 9· Match them up
Match each skew remedy to the situation it fits best
Tap a term, then the definition that fits it.
AQE splits skewed partitions only in sort merge and shuffle hash joins, and only above both thresholds. Aggregations need salting with a two-step aggregation. Broadcasting removes the shuffle that concentrated the key. REBALANCE splits skewed output partitions when AQE is on.
“If a few key values hold most of the data in a join or aggregation, append a random bucket suffix to the skewed keys”Source: docs.databricks.com
Checkpoint 9 of 9· Exam question
In the Spark UI, a job's shuffle stage shows 199 tasks finishing in under 10 seconds while a single task runs for 25 minutes and dominates the stage's wall-clock time. The stage performs a groupBy("customer_id").agg(sum("amount")) where one customer_id accounts for a large share of all rows. What is this symptom, and which change directly addresses its root cause?
Correct answer: A — This is data skew from an overrepresented key; salt the skewed key with a random suffix, aggregate on the salted key, then aggregate the partial results again to combine them.
- A. One massively overrepresented key sending nearly all rows to a single reduce task is the textbook definition of data skew, and salting spreads that key's rows across many synthetic sub-keys before a second aggregation recombines them, which is the standard fix that attacks the uneven data distribution directly.
- B. More executor memory can help a task that is failing with out-of-memory errors, but it does not rebalance how many rows land on each task, so the one oversized partition would still take far longer than the others.
- C. Compacting small files improves read throughput and file listing overhead, but it has no effect on how shuffle output is partitioned by aggregation key, so the skewed key would still dominate a single task.
- D. There is no broadcast exchange in this plan since
groupBywith an aggregation triggers a shuffle, not a broadcast, so raising the broadcast timeout has no bearing on this stage's runtime at all. - E. Trimming unused columns can reduce shuffle payload size somewhat, but it does not change which rows are routed to which reduce task, so the key causing the imbalance would still overwhelm one task.
Sources
(12)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/optimizations/aqeOfficial docs
“Very small tasks have worse I/O throughput and tend to suffer more from scheduling overhead and task setup overhead.”
↩︎ Partition count as the utilization lever“The default number of partitions to use when shuffling data for joins or aggregations.”
↩︎ Partition count as the utilization lever“The current or final plan has a different physical join node than the initial plan.”
↩︎ Reducing shuffles and confirming it in explain()“AQE treats a partition as skewed only when both of these conditions are true:”
↩︎ Identifying and fixing data skew“The default number of partitions to use when shuffling data for joins or aggregations.”
↩︎ Exam trap 1 - 2.
“Spill happens when Spark runs low on execution memory, which Spark uses for shuffles, joins, sorts, and aggregations.”
↩︎ Partition count as the utilization lever“If the Max duration exceeds the 75th percentile by 50% or more, the stage might have skew.”
↩︎ Identifying and fixing data skew“The stage details at the top of the stage page show spill statistics when the stage has spill”
↩︎ Identifying and fixing data skew“Broadcast joins are never skew-optimized”
↩︎ Identifying and fixing data skew“append a random bucket suffix to the skewed keys, and then group and aggregate in two stages”
↩︎ Identifying and fixing data skew“Spark ignores the hint if AQE isn't enabled.”
↩︎ Identifying and fixing data skew“The other cores sit idle while those tasks finish, which results in poor cluster utilization and longer jobs.”
↩︎ Key concept“Skew happens when one or a few tasks take much longer than the rest.”
↩︎ Exam trap 4“If a few key values hold most of the data in a join or aggregation, append a random bucket suffix to the skewed keys”
↩︎ Checkpoint - 3.https://www.databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.htmlSecondary source
“If there are too few partitions, then the data size of each partition may be very large”
↩︎ Partition count as the utilization lever“Shuffle is an expensive operator as it needs to move data across the network”
↩︎ Reducing shuffles and confirming it in explain() - 4.https://spark.apache.org/docs/latest/sql-performance-tuning.htmlSecondary source
“You do not need to set a proper shuffle partition number to fit your dataset.”
↩︎ Partition count as the utilization lever“they can be used for performance tuning and reducing the number of output files”
↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints“Spark SQL can cache tables using an in-memory columnar format”
↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints“Timeout in seconds for the broadcast wait time in broadcast joins”
↩︎ Identifying and fixing data skew“Configures the number of partitions to use when shuffling data for joins or aggregations.”
↩︎ Checkpoint - 5.
“can be an int to specify the target number of partitions or a Column.”
↩︎ Partition count as the utilization lever“Returns a new DataFrame partitioned by the given partitioning expressions. The resulting DataFrame is hash partitioned.”
↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints - 6.
“this operation results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle”
↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints“this may result in your computation taking place on fewer nodes than you like”
↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints“This will add a shuffle step, but means the current upstream partitions will be executed in parallel”
↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints“If a larger number of partitions is requested, it will stay at the current number of partitions.”
↩︎ Exam trap 2“If a larger number of partitions is requested, it will stay at the current number of partitions.”
↩︎ Prediction“To avoid this, you can call repartition(). This will add a shuffle step, but means the current upstream partitions will be executed in parallel”
↩︎ Checkpoint“This will add a shuffle step, but means the current upstream partitions will be executed in parallel”
↩︎ Checkpoint - 7.https://docs.databricks.com/aws/en/pyspark/reference/classes/dataframe/repartitionByRangeOfficial docs
“Due to performance reasons this method uses sampling to estimate the ranges.”
↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints - 8.https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-qry-select-distributebyOfficial docs
“Persons with same age are clustered together.”
↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints“Persons with same age are clustered together.”
↩︎ Exam trap 3 - 9.
“COALESCE, REPARTITION, and REPARTITION_BY_RANGE hints are supported and are equivalent to coalesce, repartition, and repartitionByRange Dataset APIs, respectively.”
↩︎ repartition() vs coalesce(), by column, by range, and as SQL hints“Exchange RoundRobinPartitioning(100), false, [id=#121]”
↩︎ Reducing shuffles and confirming it in explain() - 10.
“add filters before the join to reduce input rows”
↩︎ Reducing shuffles and confirming it in explain()“It chose a broadcast join instead of shuffling data across the cluster”
↩︎ Reducing shuffles and confirming it in explain()“Run OPTIMIZE to compact small files.”
↩︎ Identifying and fixing data skew - 11.
“Marks a DataFrame as small enough for use in broadcast joins.”
↩︎ Reducing shuffles and confirming it in explain() - 12.
“+- Exchange hashpartitioning(k#33, 200), true, [id=#59]”
↩︎ Reducing shuffles and confirming it in explain()