---
title: "Spark | Skills We Assign For | Azendo"
description: "Apache Spark for distributed processing — shuffles, skew, and when a cluster is not needed, plus the Azendo roles assigned for it."
url: "https://azendo.co/skills/spark/"
---

[Skills](https://azendo.co/skills/) Data pipelines 

# Spark.

Apache Spark is a distributed processing engine for large-scale data, executing transformations across a cluster with an optimiser that plans the work. It covers batch, streaming, SQL and machine learning under one API.

## Where Spark fits on a long engagement.

Spark performance is mostly a conversation about shuffles. Operations that need data grouped by key move it across the network, and that movement dominates runtime on any non-trivial job. Reducing shuffles, and making the ones that remain cheaper, is where nearly all real optimisation happens.

Data skew is the failure that looks like a hang. If one key holds a disproportionate share of rows, one task does most of the work while the cluster idles, and the job appears stuck at 99% for an hour. It is diagnosed by looking at task-level distribution, which is not where most people look first.

## What an assigned team does with Spark.

The most valuable Spark advice is often that a cluster is unnecessary. Data that fits comfortably in memory on one large machine is processed faster and far more cheaply by a single-node tool, and a surprising amount of production Spark is running on datasets that would.

Saying that rather than optimising the cluster is the honest answer, and it is the kind of judgement that comes from the assigned specialists knowing the actual data volumes. That continuity is what a [dedicated software development team](https://azendo.co/services/dedicated-development-team/) provides.

## What we use Spark for.

* Shuffle-heavy jobs made tractable Partitioning and broadcast joins applied where the network movement is the cost.
* Skew diagnosed rather than waited out Task-level distribution examined, so a stuck job is explained instead of restarted.
* Right-sizing the tool to the data Single-node processing where the dataset does not justify a cluster.

## How Spark capacity is assigned.

Spark capacity is assigned under [managed data services](https://azendo.co/services/data-engineering/), with cluster necessity questioned before cluster tuning.

## Roles we assign Spark for

* [Data Engineer Data engineering](https://azendo.co/services/data-engineering/data-engineer/)

## Service lines it sits in

* [Data engineering](https://azendo.co/services/data-engineering/)

Capacity is agreed as a committed monthly capacity across a discipline, not per skill.

## Related in data pipelines

* [Talend — skill we assign for](https://azendo.co/skills/talend/)
* [SSIS — skill we assign for](https://azendo.co/skills/ssis/)
* [Change data capture — skill we assign for](https://azendo.co/skills/change-data-capture/)
* [incremental loads — skill we assign for](https://azendo.co/skills/incremental-loads/)
* [data modelling — skill we assign for](https://azendo.co/skills/data-modelling/)
* [dimensional design — skill we assign for](https://azendo.co/skills/dimensional-design/)
* [star schema and dimensional modelling — skill we assign for](https://azendo.co/skills/star-schema-and-dimensional-modelling/)
* [slowly changing dimensions — skill we assign for](https://azendo.co/skills/slowly-changing-dimensions/)

## Tell us what your roadmap needs Spark for.

A service delivery manager replies with the disciplines we would assign, the monthly capacity and what the first month looks like.

[All skills we assign for](https://azendo.co/skills/)