Real-time Inference Endpoints

Deploying a model for synchronous predictions on Amazon SageMaker AI follows five steps.

1. Prepare the model

Package the trained model with its inference code. The artefact is a model.tar.gz containing the model files and an inference.py script defining how input is deserialised, how the model is loaded, how prediction runs and how output is serialised.

2. Create the SageMaker AI model

Register the model with the CreateModel API or in the console, specifying the container image to run and the S3 location of the model artefacts.

3. Create an endpoint configuration

Define the serving hardware:

• Instance type
• Instance count
• Production variants, if more than one model version is to receive traffic

4. Create the endpoint

Deploy using the endpoint configuration and wait for the status to become InService. The endpoint name is what callers reference.

5. Invoke the endpoint

Call InvokeEndpoint with correctly formatted input and the matching content type. Predictions come back in the response body.

Minimum deployment

Using the SageMaker Python SDK:

# Basic deployment flow
model = Model(
    model_data='s3://bucket/model.tar.gz',
    image_uri=container_uri,
    role=role)

predictor = model.deploy(
    instance_type='ml.m5.large',
    initial_instance_count=1)

Before choosing a real-time endpoint

A real-time endpoint bills for its instances continuously, whether or not it is receiving traffic. That is the right trade only when predictions genuinely have to be returned inside a request.

• Predictions over a whole dataset at once — use batch transform, which provisions compute for the job and releases it
• Intermittent traffic that can tolerate a cold start — use serverless inference, which scales to zero
• Large payloads or long processing times that do not need an interactive response — use asynchronous inference
• Many models, each lightly used — use a multi-model endpoint rather than one endpoint each

For real-time endpoints that do stay up, attach autoscaling so capacity follows traffic rather than sitting at peak, and delete endpoints created for experiments — a forgotten endpoint is the most common source of unexpected SageMaker AI spend.