SAP-C02
Deep Dive
Petabyte-scale data architectures are a frequent topic on the SAP-C02. The exam expects you to be able to design end-to-end data pipelines: ingestion, transformation, storage, querying and governance.
Contents
Ingest
Capturing data from multiple sources: batch and streaming.
• Kinesis Data Streams / Amazon Data Firehose — real-time streaming
• AWS DMS — migrating transactional databases
• AWS DataSync — transferring on-premises data to S3
• S3 Transfer Acceleration — fast uploads from global locations
• Direct Connect / AWS Snow Family — bulk transfers
Raw Storage
S3 as the central repository. Data is stored in its original format.
• S3 (raw/landing zone) — untransformed data in its original format
• Glacier — cold or long-term archived data
• S3 Intelligent-Tiering — automatic lifecycle for unpredictable access patterns
Process
Transformation, cleansing and enrichment of data.
• AWS Glue — serverless ETL with a data catalog
• EMR — Spark/Hadoop for massive processing with full control
• Lambda — lightweight serverless transformation
• Managed Service for Apache Flink — streaming processing
Curated Storage
Transformed data ready for analytical consumption.
• S3 (curated zone) — Parquet/ORC for efficient queries with Athena
• Redshift — data warehouse for OLAP and BI
• DynamoDB — low-latency access for real-time dashboards
• ElastiCache — cache for frequent query results
Consume
Visualization, ML and access to the processed data.
• Athena — serverless SQL over S3
• Quick Sight — BI and dashboards (native AWS, formerly QuickSight)
• SageMaker — training ML models on lake data
• Redshift Spectrum — queries over S3 from Redshift
• OpenSearch — log search and analysis
Amazon Redshift is AWS's managed columnar data warehouse, built for OLAP (Online Analytical Processing). It scales to petabytes and supports complex SQL queries over aggregated data.
Architecture components
Key features for the exam
Glue Data Catalog
Central metadata repository: schemas, tables, partitions. Compatible with Hive Metastore. Used by Athena, Redshift Spectrum, EMR and Glue jobs to discover data.
Glue Crawlers
Automatically scan data sources (S3, RDS, DynamoDB) and create/update table definitions in the Data Catalog. No manual schema definition code needed.
Glue ETL Jobs
Python/Scala scripts that transform data using managed Apache Spark. Auto-generated code or your own scripts. Spark or Python Shell mode (for small workloads).
Glue DataBrew
Visual tool for data cleaning and normalization without code. 250+ pre-built transformations. For non-technical users.
Glue Streaming ETL
Processes data from Kinesis Data Streams or MSK (Kafka) in micro-batches with Spark Structured Streaming. For real-time data pipelines.
Glue Workflows
Orchestrates multiple Glue jobs and crawlers in a DAG (directed acyclic graph). Manual, scheduled or event-based triggers.
Glue vs EMR — when to use each
Use Glue when:
Use EMR when:
Lake Formation centralizes Data Lake security. Instead of managing S3 permissions directly, Lake Formation defines table- and column-level permissions in the Data Catalog, which are enforced across all services that use it (Athena, Redshift Spectrum, Glue).
Column-level security
Restricts access to specific columns (e.g., hiding PII like SSN or email from certain roles).
Row-level security
Filters rows based on the user running the query (e.g., each analyst only sees data from their own region).
Cell-level security
Combines row and column for maximum granular control.
Data Filters
Conditions automatically applied to queries without the user seeing them.
Cross-account access
Share Data Catalog tables with other AWS accounts via Lake Formation permissions.
Tag-based access
LF-Tags: permissions based on tags instead of specific ARNs. Scales to thousands of tables.
Key features
Cost optimization in Athena
OpenSearch is the successor to Amazon Elasticsearch Service. It's a distributed search and analytics engine for logs, documents and metrics. The exam mentions both names.
Full-text search
Search over JSON documents with relevance scoring. Ideal for product catalogs, technical documentation.
Log analysis
Kibana/OpenSearch Dashboards for visualizing application logs, CloudWatch Logs via Subscription Filters.
Geospatial search
Proximity and map queries. For geolocation apps and fleet tracking.
| Workload type | Service | Reason |
|---|---|---|
| Ad-hoc SQL queries over data in S3 | Athena | Serverless, pay-per-scan, no cluster to maintain |
| Data warehouse for BI with fact/dimension tables | Redshift | Columnar, complex joins, BI connectors, concurrency scaling |
| Batch ETL without managing Spark | Glue | Serverless ETL, integrated catalog, auto-generated code |
| Massive ETL with full control over Spark | EMR | Spot instances, custom configuration, cheaper at scale |
| Full-text search or log analysis | OpenSearch | Search engine with scoring, Kibana for visualization |
| Real-time stream analysis with SQL | Managed Service for Apache Flink | Flink SQL or Flink applications over continuous streams |
| ML over lake data | SageMaker + S3/Athena | Accesses S3 directly or via Athena for features |
| Column-level governance and security for the lake | Lake Formation | Centralized column/row/cell security for Athena + Redshift |
Trap: "Redshift is for OLTP (transactions)"
Reality: FALSE. Redshift is OLAP (analytical). For OLTP use RDS or Aurora. Redshift isn't optimized for frequent single-row writes.
Trap: "Athena stores the data it queries"
Reality: FALSE. Athena queries data that already lives in S3. It doesn't move or store data — it only reads it. Storage lives in S3.
Trap: "Glue Crawlers are required to use Athena"
Reality: They aren't mandatory. You can define tables manually in the Data Catalog. Crawlers just automate that process.
Trap: "EMR is serverless like Glue"
Reality: FALSE. EMR requires cluster provisioning (EC2). You can use spot instances to reduce cost, but it isn't serverless. EMR Serverless exists for processing without managing clusters.
Did you get this topic?
Test what you just learned
A company needs to build a Data Lake in AWS. Data arrives via streaming from IoT devices (100K events/sec), gets processed with complex transformations, and business analysts need to run ad-hoc SQL queries. The security team requires that certain analysts NOT be able to see the 'customer_id' column. What is the correct architecture?
Put what you learned to the test with original SAP-C02 exam questions.
Start SAP-C02 mock examUnlimited mock exams with the Pro plan
Log in to track your progress.