EC2 Auto Scaling Log Collection Solutions Comparison

Instances in an Auto Scaling group are disposable, and their local disks go with them. Anything written to /var/log on an instance that scales in is lost unless it left the host first. There are two established ways to arrange that, and they answer different questions.

Stream continuously with the CloudWatch agent

The instance runs the CloudWatch agent, which tails the log files and ships lines to CloudWatch Logs as they are written. Nothing needs to happen at termination time because nothing is left behind.

Architecture: Auto Scaling group → CloudWatch agent on each instance → CloudWatch Logs → optionally exported to S3.

Setup: install the agent from user data or bake it into the AMI, supply an agent configuration, set a retention policy on the log group, and add an S3 export or subscription filter if logs need to go elsewhere.

{
  "agent": { "run_as_user": "root" },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/application/*.log",
            "log_group_name": "/ec2/application",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 30
          }
        ]
      }
    },
    "force_flush_interval": 5
  }
}

Strengths: near-real-time; retries and buffering are built in; no dependency on a graceful shutdown, so logs survive an instance that fails outright or is reclaimed as a Spot Instance; one searchable place for the whole fleet; alarms and metric filters come for free.

Costs and limits: you pay for ingestion and storage, and again if you also export to S3; log lines are pushed as text, so heavy structuring has to happen on the instance or downstream; there is a short streaming delay; CloudWatch Logs has account-level quotas that a large fleet can reach.

Drain on termination with a lifecycle hook

A scale-in lifecycle hook holds the instance in Terminating:Wait while something copies its logs off, then releases it.

Architecture: Auto Scaling group lifecycle hook → EventBridge rule → Lambda → Systems Manager Run Command on the instance → S3 → complete-lifecycle-action.

Strengths: complete control over what is collected and how it is transformed before it lands; logs go straight to S3 at S3 prices with no ingestion charge; arbitrary processing, compression or redaction can happen in the same step; the pattern extends to anything else that must run before an instance disappears.

Costs and limits: five moving parts, each of which can fail; the instance stays alive — and billable — until the copy completes or the hook times out, which delays scale-in; a failed copy loses the logs unless you build the recovery path yourself; a host that dies without warning, or a Spot Instance reclaimed inside the two-minute notice, never reaches the hook at all.

Choosing

Stream with the CloudWatch agent for production, for anything regulated that needs an audit trail, for large fleets, and wherever the team has more services than operators. It is the lower-risk default precisely because it does not depend on an orderly shutdown.

Reach for the lifecycle hook when logs need processing that cannot happen on the instance, when CloudWatch Logs ingestion cost at your volume is genuinely prohibitive, or when a compliance requirement dictates the destination and format. In practice many estates run both: streaming for the application log, and a hook for the large artefacts — heap dumps, core files, test output — that are too big to stream and only interesting when something went wrong.

Common to both

Both need an IAM role on the instance profile scoped to exactly what they write to, encryption on the destination, and access logging on it. The agent additionally needs log group permissions and, for a central logging account, a cross-account destination. The hook additionally needs monitoring of its own: alarm on hook timeouts and on Run Command failures, because a silently broken hook looks exactly like a working one until you go looking for a log.