Analytics and observability for an event-driven system

An event-driven system produces two kinds of data that people will want to look at: operational signals, which tell you whether it is working, and business records, which tell you what it did. They have different consumers, different latency requirements and different storage economics, and trying to serve both from one store usually serves neither well.

This page is a reference design that separates them while sharing a single capture path. It is deliberately generic — substitute your own domain for the transaction and token entities.

Start with a capture pipeline, not a warehouse

The most common mistake in this area is designing the full analytics platform before there is anything to analyse. A minimum viable version captures everything from day one and defers the warehouse:

flowchart TB subgraph App["Application"] API["API Gateway + Lambda"] DDB[("DynamoDB<br/>transactions, entities, audit")] EB["EventBridge / SNS / SQS"] end subgraph Capture["Capture layer"] KDS["Kinesis Data Streams"] FH["Amazon Data Firehose"] S3R["S3 — raw zone"] GLUE["AWS Glue ETL"] S3P["S3 — curated zone"] end subgraph Report["Reporting"] ATH["Athena"] QS["QuickSight"] RAPI["Report API<br/>API Gateway + Lambda"] end Clients(("External clients<br/>and partners")) API --> DDB API --> EB DDB -->|DynamoDB Streams| KDS EB --> KDS KDS --> FH FH --> S3R S3R --> GLUE GLUE --> S3P S3P --> ATH ATH --> QS S3P --> RAPI Clients --> RAPI

Application layer. API Gateway and Lambda, with DynamoDB as the operational store. Turn on DynamoDB Streams and point-in-time recovery from the beginning; retrofitting change capture onto a table that has been running for a year means you have already lost the year.

Capture layer. DynamoDB Streams and the event bus both feed Kinesis Data Streams; Amazon Data Firehose buffers and writes to S3. Partition the raw zone by date so Athena can prune partitions, enable versioning, and set lifecycle rules to move older partitions to colder storage classes.

Note the service name: Amazon Data Firehose. It was renamed from Kinesis Data Firehose in February 2024, and a lot of older material — and a lot of infrastructure code — still uses the old name.

Processing. AWS Glue for scheduled batch transformation into a curated zone; Lambda for light per-record enrichment. Keep the raw zone immutable so that a transformation bug is recoverable by reprocessing rather than by asking the source system for the data again.

Reporting. Athena queries the curated zone directly; QuickSight builds on Athena for dashboards and scheduled reports; a small API on top of the same data serves external clients and partners who want to pull rather than be shown.

This gets you complete capture and workable reporting for a fraction of the effort of a warehouse, and the warehouse — if it is ever justified — is then built from data you already have.

Classify data by temperature, not by table

Storage cost, access latency and retention are the three axes, and they correlate:

Tier	Contents	Access pattern	Typical store
Operational (hot)	Live transactions, active entities, current state	Millisecond, high volume	DynamoDB, Aurora
Analytical (warm)	Aggregates, time series, derived metrics	Seconds, query-shaped	Curated S3 with Athena, or a warehouse
Reference	Configuration, categories, fee structures, directories	Read-mostly, small	The operational store, versioned
Archive (cold)	Completed transactions, inactive entities, closed accounts	Rare, tolerant of latency	S3 Glacier storage classes

Two points that are easy to get wrong:

• Retention is a requirement, not a default. Every tier needs a stated retention period, and for regulated data that period comes from the applicable regime and your own legal advice — not from an architecture document. Design the mechanism (lifecycle rules, TTL attributes, an archive job) so that whatever period is set can be applied and evidenced.
• Reference data needs history. Fee structures and category mappings change, and a report over last quarter must use last quarter’s version. Version reference data rather than updating it in place.

Alongside the tiers, maintain a data dictionary that records, for each element, its technical name, business meaning, type and format, source system, security classification, retention period and owner. Lineage tooling — AWS Glue Data Catalog, or a dedicated catalogue product — tracks where each element came from and what depends on it; the value shows up the first time someone asks whether a number can safely change.

Observability for the operators

The operational half of the picture uses different services and a different latency budget.

flowchart LR subgraph Sources["Sources"] L["Lambda functions"] EB["EventBridge"] DDB["DynamoDB"] end subgraph Signals["Signals"] CW["CloudWatch<br/>metrics and logs"] XR["X-Ray<br/>traces"] CT["CloudTrail<br/>API activity"] end subgraph Views["Views"] GRAF["Amazon Managed Grafana<br/>operational dashboards"] QS["QuickSight<br/>business dashboards"] S3["S3 data lake"] end L --> CW L --> XR EB --> CW DDB --> CW L --> CT CW --> GRAF XR --> GRAF CT --> S3 CW --> S3 S3 --> QS

Correlation IDs are the foundation. Generate one at the edge if the request did not bring one, put it in every structured log line, and propagate it through every event and every downstream call. Without it, a choreographed system produces logs that cannot be assembled into a story.

def lambda_handler(event, context):
    correlation_id = event.get("correlation_id", str(uuid.uuid4()))
    logger.info({
        "correlation_id": correlation_id,
        "event": "PROCESS_START",
        "timestamp": datetime.now(timezone.utc).isoformat(),
    })

Annotate traces with business dimensions. X-Ray annotations are indexed and filterable, which turns “why is this slow” into a question you can answer by dimension rather than by guesswork:

const AWSXRay = require('aws-xray-sdk');

exports.handler = async (event) => {
    const subsegment = AWSXRay.getSegment().addNewSubsegment('ProvisioningFlow');
    try {
        subsegment.addAnnotation('Provider', event.provider);
        subsegment.addAnnotation('FlowType', 'Provisioning');
        subsegment.addMetadata('requestContext', {
            correlationId: event.correlationId,
            requestTime: new Date().toISOString(),
        });
        // ... flow steps, each as its own subsegment
    } catch (error) {
        subsegment.addError(error);
        throw error;
    } finally {
        subsegment.close();
    }
};

Keep annotations low-cardinality — provider, flow type, status — and put anything high-cardinality, such as the correlation ID itself, in metadata.

Emit business metrics explicitly. Infrastructure metrics tell you the platform is healthy; they will not tell you that success rates fell for one provider. Publish the handful of counters and timings that describe the business outcome, with dimensions you will actually filter on.

Derive metrics from your own structured logs, using CloudWatch metric filters over the events the application emits:

MetricFilters:
  ProvisioningSuccess:
    FilterPattern: '{ $.event = "PROVISION_SUCCESS" }'
    MetricNamespace: "Provisioning"
    MetricName: "SuccessfulProvisioning"
    MetricValue: "1"

  ProvisioningLatency:
    FilterPattern: '{ $.event = "PROVISION_SUCCESS" }'
    MetricNamespace: "Provisioning"
    MetricName: "ProvisioningLatency"
    MetricValue: "$.processingTime"

Alarms:
  HighFailureRate:
    MetricName: "FailedProvisioning"
    Namespace: "Provisioning"
    Period: 300
    EvaluationPeriods: 2
    Threshold: 5
    ComparisonOperator: "GreaterThanThreshold"

Note that these filter on the application’s own JSON log events, not on CloudTrail. CloudTrail records AWS API calls made by real AWS services — it will show the DynamoDB and Lambda calls underneath a flow, and it is the right source for an audit trail of infrastructure activity, but it does not and cannot carry your domain events. Build the business-flow metrics from your own logs and keep the two audit trails separate.

Choosing the dashboard

	Amazon Managed Grafana	Amazon QuickSight
Audience	Engineers and operators	Business users and executives
Latency	Seconds to minutes	Minutes to hours
Sources	CloudWatch, X-Ray, Prometheus, OpenSearch and many others	Athena, S3, RDS, Redshift, files
Strengths	Real-time panels, service maps, sophisticated alerting	ML-driven insights, embedding, per-reader pricing, sharing outside the engineering team

Most estates end up running both: Grafana for the operational view over CloudWatch and X-Ray, QuickSight for the business view over the curated S3 zone, sharing the data lake underneath. If you support that split, name the boundary explicitly — which dashboards are authoritative for which questions — or the two will disagree at the worst moment and nobody will know which to believe.

One naming correction worth carrying into infrastructure code: the service is Amazon Managed Grafana, not AWS Managed Grafana.

And one service to avoid selecting: Amazon Timestream for LiveAnalytics closed to new customers on 20 June 2025. Older reference architectures — including AWS’s own, for a while — routed time-series metrics there. If you are building today, use CloudWatch metrics (with metric streams into a longer-term store where you need retention beyond CloudWatch’s) or Amazon Timestream for InfluxDB, which AWS recommends as the equivalent for new workloads.

Security and access

• Role-based access to reports, with tenant data segregated at the query layer and not merely in the dashboard.
• Encryption at rest and in transit throughout; a customer-managed KMS key where key policy or rotation matters.
• Audit logging of report access, not just of data changes. Who read the export is a question that gets asked after the fact and cannot be answered retrospectively.
• Masked or synthetic data in non-production environments. A copy of production in a development account is the most common way sensitive data escapes its controls.

Reference

• Amazon Data Firehose
• Amazon Timestream for LiveAnalytics availability change
• CloudWatch Logs metric filters