All guides
Practice now
D3 · Continuous Improvement for Existing Solutions

Advanced observability: CloudWatch, X-Ray and Container Insights

Observability is the ability to understand a system's internal state from its external outputs. On the SAP-C02, observability questions go beyond basic — they test complete monitoring architectures for distributed systems at scale.

Icon-Architecture/48/Arch_Amazon-CloudWatch_48

The 3 pillars of observability on AWS

📊

Metrics

Numeric data aggregated over time. How much? How fast? How many errors?

  • • CloudWatch Metrics (native and custom)
  • • CloudWatch Embedded Metrics (EMF)
  • • Container Insights
  • • Lambda Insights
📝

Logs

Event records with full context. What exactly happened, and when?

  • • CloudWatch Logs
  • • CloudWatch Logs Insights (queries)
  • • Log Subscriptions to Kinesis/Lambda
  • • S3 export for analysis with Athena
🔍

Traces

Following requests across multiple services. Where's the ?

  • • AWS X-Ray
  • • CloudWatch ServiceLens (combines traces + metrics + logs)
  • • AWS Distro for OpenTelemetry (ADOT)
Icon-Architecture/48/Arch_Amazon-CloudWatch_48

Advanced CloudWatch — beyond basic monitoring

Advanced metrics

Custom MetricsPublished via the PutMetricData API. Granularity: 1s (high resolution) or 1 min. For business metrics (orders per minute).
Metric MathMathematical operations on metrics: ratios, sums, error rates. E.g.: ErrorRate = Errors / Requests * 100.
Anomaly DetectionAn ML model that learns the historical pattern of a metric and alerts when it deviates — no static thresholds to define.
Composite AlarmsCombine multiple alarms with AND/OR logic. Reduces "alarm noise" — only alerts when multiple conditions are met simultaneously.
Metric StreamsReal-time metric streaming to Amazon Data Firehose → S3 or third-party destinations (Datadog, New Relic).

Advanced CloudWatch Logs

Log InsightsIts own query language for analyzing structured logs. Faster and cheaper than exporting to Athena for ad-hoc queries.
SubscriptionsReal-time log streams to Lambda or Kinesis for processing or transformation.
Log Groups RetentionConfigurable from 1 day to forever. Without a retention setting, logs never expire (indefinite cost).
Cross-Account SharingSend logs from member accounts to a centralized logging account via Subscription Filters + Kinesis.
Contributor InsightsIdentifies the "top contributors" in your logs — e.g., the IPs with the most 429 errors, the slowest URLs.
Icon-Architecture/48/Arch_Amazon-CloudWatch_48

Container Insights and Lambda Insights

Container Insights

Collects metrics and logs from containers on , EKS, and Kubernetes. Aggregates data at the cluster, node, pod, and individual container level.

  • CPU/Memory/Network/Storage per container, pod, and node
  • ECS: metrics for Tasks, Services, and Clusters
  • EKS: integrated via CloudWatch Agent as a DaemonSet
  • Performance logs in CloudWatch Logs for detailed diagnostics
  • Requires CloudWatch Agent or Fluent Bit on worker nodes

Lambda Insights

Advanced performance monitoring for functions. Diagnoses duration, throttling, and memory error issues.

  • Total CPU time, memory used vs allocated, init duration
  • Detects functions with frequent out-of-memory errors
  • Network bytes sent/received per invocation
  • Enabled as a Lambda Layer — no code changes needed
  • Data in CloudWatch Logs using EMF (Embedded Metrics Format)
Icon-Architecture/48/Arch_AWS-X-Ray_48

AWS X-Ray — distributed tracing

X-Ray generates a visual map of your microservices architecture showing how requests flow between services, where the latency is, and what errors occur. Essential for debugging distributed systems.

Key X-Ray concepts

TraceA collection of segments that together represent an end-to-end request (from the client through every service).
SegmentThe unit of work for a single service. Contains timing, errors, and request metadata.
SubsegmentDetailed work within a segment: a call to , a SQL query, an external HTTP call.
Sampling RulesDefine what percentage of requests get traced. Default: 5% + 1 req/sec per host. Configurable per service, URL, HTTP method.
X-Ray DaemonA process that receives segments from the app via UDP and forwards them to the X-Ray API. Needed on EC2 and ECS (not on Lambda, where it is built in).

Integration with AWS services

Lambda

Native — enable Active Tracing on the function

Native — enable X-Ray Tracing on the stage

ECS/EC2

Requires the X-Ray daemon as a sidecar or daemon

EKS

ADOT Collector as a DaemonSet or sidecar

ECS Service Connect

The managed proxy publishes traffic metrics automatically

Elastic Beanstalk

Native — configure via .ebextensions

CloudWatch ServiceLens

ServiceLens combines X-Ray traces, CloudWatch metrics, and CloudWatch logs into a unified view. It generates a Service Map showing the health of each service with latency, error, and request-rate metrics — no need to jump between multiple consoles.

Icon-Architecture/48/Arch_Amazon-CloudWatch_48

Feature flags and A/B testing: AWS AppConfig (CloudWatch Evidently was retired)

CloudWatch Evidently was retired on October 17, 2025. Today feature flags are managed with AWS AppConfig (Systems Manager): it lets you roll out features gradually, validate configuration, and roll back automatically if a CloudWatch alarm fires. A/B experiments are built by combining the flags with your own business metrics.

Feature Flags

AppConfig enables features for a % of users without redeploying code. Enables controlled canary releases.

A/B Experiments

AppConfig variants are combined with business metrics in CloudWatch to measure impact.

Automatic rollback

AppConfig reverts the configuration if a CloudWatch alarm fires during deployment.

Segmentation

Shows features based on user attributes: region, plan, app version.

Icon-Architecture/48/Arch_AWS-CloudTrail_48

AWS — auditing and governance

Event typeWhat it recordsCostUse case
Management EventsControl plane operations: creating/modifying/deleting resourcesFree (first trail)Auditing infrastructure changes — "who created this resource"
Data EventsData-plane operations: S3 GetObject, Lambda Invoke, DynamoDB GetItemPaidAuditing access to sensitive data, post-incident forensics
Insights EventsUnusual activity: spikes in IAM errors, unusual API callsPaidDetecting anomalous behavior in API calls

CloudTrail Organization Trail

An Organization Trail created in the Management Account records events from every member account into a centralized S3 bucket. Member accounts cannot modify or disable the trail. This guarantees complete, immutable auditing for the whole organization.

Icon-Architecture/48/Arch_Amazon-CloudWatch_48

Common exam pitfalls

Pitfall: "X-Ray records every request by default"

Reality: FALSE. X-Ray uses sampling — by default it records 5% of requests + 1 request/sec. The sampling rate can be increased for complete debugging.

Pitfall: "CloudWatch and CloudTrail are equivalent"

Reality: These are completely different. CloudWatch monitors performance (metrics, app logs). CloudTrail audits AWS API calls (who did what with which resources).

Pitfall: "Container Insights works with no extra configuration on EKS"

Reality: FALSE. On EKS it requires deploying the CloudWatch Agent (or Fluent Bit) as a DaemonSet. On ECS with Fargate it requires a sidecar container with the CloudWatch Agent.

Did you get this topic?

Test what you just learned

A company has a microservices architecture with 15 services on Lambda and ECS. Users report that some requests intermittently take more than 10 seconds. The operations team can't determine which specific service is causing the latency. Which AWS service directly solves this problem?

Ready to test "Advanced Observability: CloudWatch, X-Ray and Container Insights"?

Put what you learned to the test with original SAP-C02 exam questions.

Start SAP-C02 mock exam

Unlimited mock exams with the Pro plan

Log in to track your progress.