Elastic Inference (retired)

Amazon Elastic Inference attached a fraction of a GPU to a CPU instance so that inference could be accelerated without paying for a whole GPU instance. It had to be configured when the deployable model was created, and it did not support every algorithm.

Elastic Inference no longer exists. AWS stopped accepting new customers and the service reached end of life; its documentation has been withdrawn and redirects to the Amazon SageMaker AI inference guide. It cannot be attached to a new model, and any architecture document that recommends it needs revising.

What covers the same need now

Inference cost optimisation on SageMaker AI is addressed by several features rather than one.

Serverless inference — endpoints with no instance to size, which scale to zero between requests. The natural fit for the workload Elastic Inference was usually sold against: intermittent traffic that does not justify a dedicated accelerator, and that can tolerate a cold start.

Asynchronous inference — queues requests and processes them in the background, supporting large payloads and long processing times. Removes the need to provision for peak on workloads that are not interactive.

Automatic scaling — adjusts endpoint capacity to traffic, so a real-time endpoint is sized to current demand rather than to its busiest hour.

Multi-model endpoints — host many models behind one endpoint, loading each on demand. Where the alternative is dozens of lightly used endpoints, this is the largest single saving available.

SageMaker Neo — compiles a model for the specific target hardware, which reduces the compute needed per inference. See SageMaker Neo.

AWS Inferentia and Graviton instances — purpose-built inference silicon and ARM-based general-purpose instances, both offering better price-performance than equivalent x86 GPU instances for many models.

Batch transform — where predictions do not need to be served in real time at all, running them as a job removes the endpoint entirely.

Choosing between them

Work through in this order: can the predictions be batched (batch transform), can they be queued (asynchronous inference), is traffic intermittent (serverless inference), are there many models (multi-model endpoints), and only then reach for instance-level tuning with Neo, Inferentia or Graviton.