Step Function and Batch
Four AWS services are commonly proposed for “run this work in the right order”: Step Functions, Simple Workflow Service, SQS and AWS Batch. They are not interchangeable, and the choice usually follows from how much coordination the work needs and what runs it.
AWS Step Functions
Section titled “AWS Step Functions”Step Functions is a managed workflow and orchestration service for coordinating AWS components. It is highly available, scales without configuration, and keeps the coordination logic out of application code.
Key features
Section titled “Key features”- State machine definition — tasks, sequential and parallel steps, decision points and timers
- Amazon States Language — declarative JSON that both configures and documents the workflow
- Visual interface — real-time status and a flow view of each execution
- Service integrations — direct calls to Lambda and a large catalogue of AWS services without glue code
- Detailed logging — every step of every execution recorded
The state machine model
Section titled “The state machine model”Step Functions implements a finite state machine: an entity moves through defined states over its lifetime. An order-processing flow, for example:
- New order — the initial state
- Credit check — moves to cancelled or verified
- Warehouse processing — moves to shipping
- Delivery — moves to delivered
- Final processing — moves to archived or returned
Work that has to pause for a human decision is handled with the callback pattern: a task
using .waitForTaskToken suspends the execution until something calls SendTaskSuccess
or SendTaskFailure with the token — an approval in an internal tool, a reply to an
email. That pattern is available in Standard workflows only.
AWS Batch
Section titled “AWS Batch”AWS Batch runs batch computing workloads: it queues jobs, provisions compute for them, runs them and scales the compute back down. Jobs are containers, and the compute underneath them can be EC2 instances, Fargate, Amazon ECS Managed Instances or an Amazon EKS cluster — Fargate being the option that requires no instance management at all.
Putting a Batch workload together
Section titled “Putting a Batch workload together”- Define a compute environment
- Managed (Batch provisions and scales the capacity) or unmanaged (the account does)
- Choose the platform: Fargate, EC2 instances (Spot or On-Demand), ECS Managed Instances, or EKS
- Specify the vCPU range the environment may scale between
- Create a job queue
- Give it a priority and attach it to one or more compute environments
- Define a job
- A container image, a command or JSON definition, environment variables, mount points and an IAM role
- Submit the job
- Batch schedules it against the queue’s compute environments and scales capacity to match the backlog
Choosing between them
Section titled “Choosing between them”| Service | When | Example |
|---|---|---|
| Step Functions | Out-of-the-box coordination of AWS components, with branching, retries and an audit trail | Order processing flow |
| Simple Workflow Service (SWF) | Existing SWF implementations, or workflows whose logic runs in long-lived external workers outside AWS | Legacy claims processing already built on SWF |
| Simple Queue Service (SQS) | Store-and-forward: distributing independent units of work to consumers | Image resizing pipeline |
| AWS Batch | Scheduled or recurring compute-heavy jobs with little decision logic | Nightly log rotation or a rendering run |
On SWF
Section titled “On SWF”Amazon SWF still exists and is still supported, but AWS’s own guidance is to consider Step Functions for workflow and orchestration needs. The human-in-the-loop case that once justified SWF is now covered by the Step Functions callback pattern above, so SWF is best treated as the answer for estates that already run on it rather than a candidate for new work.
Step Functions or SQS
Section titled “Step Functions or SQS”A queue distributes work; it does not sequence it. If step B must follow step A, if a failure must trigger compensation, or if anyone will ask later what happened to a particular request, that is a workflow, and putting it in a chain of queues means rebuilding Step Functions by hand. If each message is independent and the only requirement is that it eventually gets processed, a queue is simpler and cheaper.
Step Functions or Batch
Section titled “Step Functions or Batch”Batch is about capacity: getting a container onto the right compute at the right scale. Step Functions is about sequence: deciding what runs next and what happens when it fails. They compose — a Step Functions task can submit a Batch job and wait for it — and a data pipeline frequently uses both.