CDL
Deep Dive
How to move, transform, and visualize data in Google Cloud. The standard GCP pipeline and BI tools.
Reference architecture
Source
Apps, IoT, Logs
Pub/Sub
Messaging
Dataflow
Transformation
BigQuery
Warehouse
Looker
Visualization
Real-time streaming pipeline. For batch: Cloud Storage → Dataflow → BigQuery
Publisher-subscriber model
Publisher
Produces messages and sends them to a Topic. It doesn't know who will read them. It can be an app, an IoT sensor, a log, etc.
Topic
Message channel. Messages are stored until they are acknowledged by all subscribers or until the retention period expires (max 7 days).
Subscription
Receiver of messages from a topic. Can be push (Pub/Sub sends) or pull (the subscriber asks). Multiple subscriptions per topic.
| Feature | Detail |
|---|---|
| Delivery guarantee | At-least-once: a message can be delivered more than once. Consumers must be idempotent or use the Exactly-once option. |
| Throughput | Up to millions of messages per second. Scales automatically with no configuration. |
| Fan-out | A topic can have multiple subscriptions — each subscriber receives all messages independently. |
| Dead Letter Topics | Messages that cannot be processed after N attempts are redirected to another topic for analysis. |
| Message ordering | With an Ordering Key, messages with the same key are delivered in order. Without a key, order is not guaranteed. |
| Retention | Messages retained for up to 7 days. Allows replaying past messages to a new subscription. |
Dataflow is the managed service for running data pipelines based on Apache Beam. The same code works for batch (historical data) and streaming (real-time data) — this is called Beam's unified programming model.
Serverless
No workers to configure. Dataflow provisions and scales automatically.
Batch + Streaming
The same Apache Beam pipeline processes historical or real-time data.
Templates
Predefined templates for common cases: Pub/Sub → BigQuery, GCS → BigQuery, etc.
Integration
Reads from Pub/Sub, Cloud Storage, BigQuery. Writes to BigQuery, Cloud Storage, Bigtable.
Dataflow vs Dataproc
Dataflow (use when)
Dataproc (use when)
Looker (Enterprise BI)
Looker Studio (formerly Data Studio)
Did you get this topic?
Test what you just learned
An IoT sensor monitoring system generates 1 million events per second. The data must be processed in real time to detect anomalies and stored in BigQuery for historical analysis. Which GCP architecture is correct?
Put what you learned to the test with original CDL exam questions.
Start CDL mock examUnlimited mock exams with the Pro plan
Log in to track your progress.