CloudWatch Logs
CloudWatch Logs stores and serves log data from AWS services, applications and
on-premises servers. Many AWS services write to it directly — Lambda, API
Gateway access logs, VPC flow logs, ECS tasks, RDS engine logs — and anything
else reaches it through the CloudWatch agent or the PutLogEvents API.
The three levels
Section titled “The three levels”- Log event — one record: a timestamp and the raw message. This is the unit that is ingested, stored and billed.
- Log stream — an ordered sequence of log events from a single source. One instance, one container, one Lambda execution environment.
- Log group — a collection of log streams that share retention, encryption, access control and metric filter settings. All the Apache streams across a fleet belong in one group. There is no limit on the number of streams in a group.
Retention, subscription filters and metric filters are configured on the log group, never on individual streams. That is the practical reason to get the grouping right: it is the only level at which you can control anything.
Getting logs in
Section titled “Getting logs in”- CloudWatch agent — the unified agent sends both metrics and logs from EC2 instances and on-premises servers.
- Native service integration — most AWS services can be pointed at a log group directly.
- API —
PutLogEventsuploads batches programmatically; the AWS CLI exposes it asput-log-events.
Retention
Section titled “Retention”By default log data is stored indefinitely. This is the single most expensive default in CloudWatch Logs, and the one most often left untouched: a log group created three years ago by a Lambda function nobody runs any more is still being charged for storage. Set an explicit retention period on every log group you create.
Retention is set per log group and applies to the whole group. When events pass their retention setting they are marked for deletion and stop counting towards storage costs; actual deletion typically takes up to 72 hours.
Log groups can also be given deletion protection, which is worth enabling on anything you have a compliance obligation to keep.
Turning logs into signals
Section titled “Turning logs into signals”- Filter patterns search for terms within log events, and are the basis of both console searching and metric filters.
- Metric filters count matching events and publish the count as a CloudWatch metric, which you can then alarm on. This is how you get an alarm on “more than N occurrences of this error string” — the log data itself is not alarmable, the metric derived from it is.
- Subscription filters stream matching events in near real time to Kinesis Data Streams, Amazon Data Firehose, Lambda or OpenSearch Service, for processing or onward storage.
- CloudWatch Logs Insights queries stored log data interactively, across log groups, in three query languages.
- Cross-account cross-Region log centralisation replicates log groups to a central account, for estates that need one place to look.