Skip to content
PuglieseWeb Engineering HandbookSystemDox Pages

CloudWatch Logs Insights

CloudWatch Logs Insights is the query engine over data already in CloudWatch Logs. It is not a separate store and it does not ingest anything of its own: log groups hold the data, Logs Insights searches, aggregates and visualises it.

The distinction matters because the two are billed differently. CloudWatch Logs charges for ingestion and storage. Logs Insights charges on the volume of uncompressed log data scanned by each query, regardless of which query language you use. A wide time range over a busy log group is expensive; the techniques below exist mostly to narrow what gets scanned.

Three query languages

Section titled “Three query languages”

Logs Insights supports three languages against the same data. Pick by the shape of the question, not by preference.

Logs Insights QL is the purpose-built language: a small set of commands (fields, filter, stats, sort, limit, parse) chained with pipes. It is the default, it has the widest feature support, and some capabilities exist only here — comparison queries against a previous time period, and the filterIndex command.

OpenSearch Piped Processing Language (PPL) pipes commands together in the same style, with a richer library of maths, string, date and conditional functions. It is the familiar choice for teams already running OpenSearch.

OpenSearch SQL is declarative — SELECT, FROM, WHERE, GROUP BY, HAVING. Its distinguishing capability is that it can JOIN across log groups and correlate with sub-queries, which the other two cannot. When a question spans two log groups and needs their records matched rather than merely concatenated, SQL is the reason to be here.

In PPL and SQL, wrap any field whose name is not purely alphabetic in backticks — `@message`, `Operation.Export`.

What it can do

Section titled “What it can do”
  • Automatic field discovery on logs from services such as Lambda, CloudTrail, Route 53 and VPC, and on any log emitted as JSON.
  • Field indexes — index fields that appear commonly in your events, and a query can skip log events known not to contain the indexed field. Less data scanned means faster results and a smaller bill, which is the main lever you have over Logs Insights cost.
  • Pattern analysis — groups results by recurring text structure, which collapses a million near-identical lines into the handful of shapes they actually take.
  • Natural language query generation — describe what you are looking for and the console generates a query with a line-by-line explanation of how it works. It can also summarise a result set in natural language.
  • Facets for grouping and filtering results interactively.
  • Surrounding logs — view 5, 10, 20, 50 or 100 lines either side of a specific record, and search within them. This is the thing you always want two minutes into an incident.
  • Saved and parameterised queries, query history, and adding a query to a CloudWatch dashboard.
  • Cross-account queries from a monitoring account against log groups in linked source accounts.
  • Query result encryption with a KMS key.

Limits worth knowing before you rely on it

Section titled “Limits worth knowing before you rely on it”
  • 100 concurrent Logs Insights QL queries per account, including queries embedded in dashboards.
  • 15 concurrent queries for OpenSearch PPL or OpenSearch SQL.
  • Queries time out after 60 minutes; results remain available for 7 days.
  • Logs Insights cannot read log events with timestamps earlier than the log group’s creation time, and cannot search data sent to CloudWatch Logs before 5 November 2018.
  • The console interface needs web sockets. Where a network policy blocks them, use the StartQuery API instead.

Choosing between this and the alternatives

Section titled “Choosing between this and the alternatives”

Logs Insights or X-Ray? They answer different questions and the choice is rarely close. Logs Insights searches records your code chose to write; X-Ray follows one request across services and shows where its time went. Reach for Logs Insights to find every occurrence of an error, extract a count or a custom metric from log text, or investigate historically. Reach for X-Ray to see service dependencies, find which hop in a distributed call chain is slow, or debug a request that touched five services. In an incident you usually use both: X-Ray to identify the failing service, Logs Insights to read what it said.

Logs Insights or OpenSearch Service? Logs Insights is the lower-effort option and requires no cluster, but it queries CloudWatch Logs and only CloudWatch Logs. Amazon OpenSearch Service earns its operational cost when you need real full-text search, sustained querying over very large volumes where per-query scanning charges add up, log data consolidated from outside AWS, or an existing Elasticsearch-shaped workflow and the dashboards that go with it.

Logs Insights or Grafana? Grafana is a visualisation and alerting layer, not a log store — its strength is a single dashboard drawing on many backends, including CloudWatch itself. If the requirement is “one pane of glass across AWS and non-AWS systems”, that is Amazon Managed Grafana sitting on top of CloudWatch, not a replacement for it. If the requirement is “query these AWS log groups”, Logs Insights is already there.

Further reading

Section titled “Further reading”