All guides
Practice now
D3 · Continuous Improvement for Existing Solutions

Reliability and continuous improvement: AWS Health, Chaos Engineering and Access Analyzer

SAP-C02's D3 domain tests the continuous improvement of existing architectures. Reliability goes beyond Multi-AZ — it means actively testing failures, analyzing failure modes, and continuously improving your security posture with Access Analyzer and Permission Boundaries.

Icon-Architecture/48/Arch_AWS-Backup_48

Reliability pillar — Well-Architected Framework

The Reliability pillar of the Well-Architected Framework defines how to design systems that recover from failures, scale dynamically, and deliver the right capacity to meet demand.

🏗️

Foundations

Managing service limits (quotas), network capacity planning. Set appropriate service limits before they're needed.

  • • Proactively request quota increases
  • • Monitor quota approach with Trusted Advisor
  • • Design for multiple AZs from the start
⚙️

Workload Architecture

Design for HA and recovery from the initial design. Circuit breakers, retries with exponential backoff, bulkheads.

  • • Eliminate single points of failure
  • • Implement health checks on every component
  • • Design for graceful degradation
🔄

Change Management

Manage workload changes with automated testing and rollback. Auto Scaling, zero-downtime deployment.

  • • Blue/Green or Canary deployments
  • • Feature flags for safe rollouts
  • • Runbooks for change procedures

Software resilience patterns for the SAP-C02

Retry with Exponential Backoff

Retry failed operations with increasing waits + random jitter. Avoids the "thundering herd" problem.

Circuit Breaker

After N consecutive failures, "open" the circuit and stop sending requests until the service recovers.

Bulkhead

Isolate resources by function. If search collapses, checkout is unaffected. Limits threads/connections per service.

Timeout

Never wait indefinitely. Define timeouts on every external call. Failing fast beats waiting.

Failure mode analysis — identifying Single Points of Failure

FMEA (Failure Mode and Effects Analysis) is the process of systematically identifying how each component can fail and what the impact would be. On AWS, that translates into reviewing the architecture looking for SPOFs.

ComponentFailure modeAWS mitigation
single instanceThe instance fails or the underlying host failsASG with a minimum of 2 instances across 2 AZs + ALB
single-AZHardware failure → hours of downtime to restoreRDS Multi-AZ — automatic failover in < 1 min
NAT Gateway in one AZIf that AZ fails, private subnets in other AZs lose internet accessOne NAT Gateway per AZ in use
concurrency throttlingA traffic spike exhausts available concurrencyReserved Concurrency + SQS buffer to smooth out spikes
hot partitionA partition receives all writes → throttlingRedesign the partition key with more cardinality or write sharding
single bucket regionRegion outage → data unreachableCRR to a secondary-region bucket for DR
DNS with a high TTLIf the ALB/endpoint changes, cached clients hit the wrong endpointLow TTL on for critical records (60-300s)
Icon-Architecture/48/Arch_AWS-Health-Dashboard_48

AWS Health — visibility into service events

Types of AWS Health events

Service Events

Outages and degradations of AWS services affecting ALL customers in a region. E.g.: "EC2 API degraded in us-east-1". Visible on the public AWS Health Dashboard (Service health view).

Account Events

Events specifically affecting YOUR account: EC2 instances scheduled for retirement, SSL certificates nearing expiration, expiring RIs. Only visible in the Your account health view of the AWS Health Dashboard.

Scheduled Changes

Scheduled AWS maintenance that will impact your resources. E.g.: "your instance on this host will be migrated on date X". You must act before the deadline.

Automation with AWS Health + EventBridge

AWS Health events publish to EventBridge, enabling automated responses.

Instance scheduled for retirementLambda: automatic snapshot + terminate + launch a new instance
RDS maintenance scheduledLambda: notify the DBA team + block changes during that window
Spot interruption warningLambda: drain connections, checkpoint work to S3
Certificate expiringLambda: trigger ACM renewal or alert the security team

Chaos Engineering on AWS — testing failures before they happen

Chaos engineering injects controlled failures into production (or pre-production) to uncover weaknesses before they occur in real situations. AWS Fault Injection Service (FIS) is the native service for this.

AWS Fault Injection Service (FIS)

  • Randomly terminates EC2 instances (like Chaos Monkey)
  • Injects latency into API calls
  • Forces an RDS Multi-AZ failover
  • Interrupts Spot instances
  • Injects HTTP errors into ALB responses
  • Drains EKS nodes
  • Safety controls: automatic stop conditions if metrics degrade too much

AWS Resilience Hub + Game Days

  • AWS Resilience Hub: defines and validates RTO/RPO targets before running experiments
  • Game Day: a planned exercise simulating a real disaster to test runbooks
  • DRP (Disaster Recovery Plan) testing: periodically execute failover to the DR region
  • Runbooks: step-by-step documentation for responding to each type of failure
  • Blameless post-mortems: analyze what failed and improve the systems, not assign blame
Icon-Architecture/48/Arch_AWS-Identity-and-Access-Management_48

IAM Access Analyzer and Permission Boundaries

IAM Access Analyzer

Analyzes resource policies (S3 buckets, IAM roles, keys, Lambda, SQS, Secrets Manager) to identify unintended external access outside the defined trust zone.

External Access FindingsDetects S3 buckets, IAM roles, or KMS keys that allow access from outside the account or organization.
Unused Access FindingsIdentifies roles, policies, permissions, and credentials unused for 90+ days. For applying active least privilege.
Custom Policy ValidationValidates IAM policies for syntax errors, overly broad permissions, and suggests fixes.
Trust zoneYou define the zone (account or org). Access from inside = OK. Access from outside = a finding to review.

Permission Boundaries

Permission Boundaries are IAM policies that define the maximum permissions an IAM entity can have, regardless of what policies get attached to it later.

Key exam use case

"Developers can create IAM roles for their Lambda functions, but they cannot escalate their own permissions beyond what they already have."

→ A Permission Boundary that limits the roles developers can create.

  • The Permission Boundary does NOT grant permissions — it only limits them
  • A user with AdministratorAccess + an S3 Permission Boundary can only do S3
  • It can be required via SCPs: "iam:CreateRole only if it includes this permission boundary"
  • Used to delegate role creation without privilege escalation
  • Icon-Architecture/48/Arch_AWS-Backup_48

    Backup strategies at organizational scale

    For the SAP-C02, backup isn't just "create snapshots" — it's a coordinated strategy of retention, recovery testing, and compliance. AWS Backup with Organizations is the enterprise pattern.

    StrategyHow to implementWhat it's for
    3-2-1 Rule3 copies, 2 different media, 1 offsite. On AWS: original + snapshot in the same region + cross-region copy.Standard backup best-practice compliance
    Immutable backupsAWS Backup Vault Lock (WORM). Backups can't be deleted before they expire.Anti-ransomware and anti-tampering protection
    Cross-account backupsA vault in a centralized backup account. Copies from every member account go to the central vault.Protecting backups from accidental or malicious deletion in the source account
    Restore testingAWS Backup Restore Testing: automates periodic restore testing of backups.Guaranteeing that backups are actually recoverable
    Tag-based backup policiesA Backup Plan that selects resources by tag (Environment=Production). Auto-includes new resources with that tag.Governance: ensuring every production resource has a backup
    Icon-Architecture/48/Arch_AWS-Backup_48

    Common exam pitfalls

    Pitfall: "IAM Access Analyzer and the IAM Credential Report do the same thing"

    Reality: They're different. Access Analyzer analyzes resource POLICIES to find unintended access from outside the trust zone. The Credential Report lists IAM users' credentials and their status (MFA, last used, expired password).

    Pitfall: "Permission Boundaries grant additional permissions"

    Reality: FALSE. Permission Boundaries ONLY limit the maximum possible permissions. They don't grant any permission by themselves. The actual permission is the INTERSECTION of the Permission Boundary and the identity policies.

    Pitfall: "AWS FIS only works in test/staging environments"

    Reality: AWS FIS can run experiments in production (with the right safeguards: stop conditions, monitoring). In fact, its main purpose is testing in production to uncover real failures that wouldn't show up in test environments.

    Did you get this topic?

    Test what you just learned

    A platform team wants to allow development teams to create their own IAM roles for Lambda functions, but needs to guarantee those roles never have more permissions than the developers themselves have. They also want to automatically detect if any IAM role created by developers allows access from external accounts. Which combination of services implements BOTH requirements?

    Ready to test "Reliability: AWS Health, Chaos Engineering and Access Analyzer"?

    Put what you learned to the test with original SAP-C02 exam questions.

    Start SAP-C02 mock exam

    Unlimited mock exams with the Pro plan

    Log in to track your progress.