Spark.

Where Spark fits on a long engagement.

Spark performance is mostly a conversation about shuffles. Operations that need data grouped by key move it across the network, and that movement dominates runtime on any non-trivial job. Reducing shuffles, and making the ones that remain cheaper, is where nearly all real optimisation happens.

Data skew is the failure that looks like a hang. If one key holds a disproportionate share of rows, one task does most of the work while the cluster idles, and the job appears stuck at 99% for an hour. It is diagnosed by looking at task-level distribution, which is not where most people look first.

What an assigned team does with Spark.

The most valuable Spark advice is often that a cluster is unnecessary. Data that fits comfortably in memory on one large machine is processed faster and far more cheaply by a single-node tool, and a surprising amount of production Spark is running on datasets that would.

Saying that rather than optimising the cluster is the honest answer, and it is the kind of judgement that comes from the assigned specialists knowing the actual data volumes. That continuity is what a dedicated software development team provides.

What we use Spark for.

• Shuffle-heavy jobs made tractable Partitioning and broadcast joins applied where the network movement is the cost.
• Skew diagnosed rather than waited out Task-level distribution examined, so a stuck job is explained instead of restarted.
• Right-sizing the tool to the data Single-node processing where the dataset does not justify a cluster.

How Spark capacity is assigned.

Spark capacity is assigned under managed data services, with cluster necessity questioned before cluster tuning.