Prefect.

Where Prefect fits on a long engagement.

Dynamic workflow structure is Prefect's practical advantage. A flow whose shape depends on runtime data — processing however many files arrived, fanning out across whatever partitions exist — is ordinary code here, where statically defined DAGs require workarounds.

Because flows are plain Python, adoption cost is low for a team that already writes Python, and testing is ordinary function testing. The trade-off is fewer opinions: Prefect will not stop a team building an unmaintainable workflow, where a more prescriptive orchestrator would.

What an assigned team does with Prefect.

Retry configuration is where reliability is actually won. A transient API failure should retry with backoff; a malformed input should not retry at all. Treating every failure the same either masks real problems or gives up on recoverable ones.

Getting that distinction right depends on knowing the specific sources and their behaviour, which is product knowledge that accumulates in a standing assignment and is set out in how the monthly fee is built.

What we use Prefect for.

• Workflows whose shape depends on the data Fan-out across whatever arrived, without contorting a static DAG definition.
• Retries that distinguish failure types Backoff for transient errors, immediate failure for bad input.
• Flows tested as ordinary functions Logic verified without standing up a scheduler.

How Prefect capacity is assigned.

Prefect work is assigned under data engineering outsourcing, with failure handling designed per source rather than applied uniformly.