BigQuery, Pub/Sub, Dataflow, and Kafka

• Separate storage and compute: each layer scales without affecting the other, thanks to Google's network.
• Columnar storage optimized for analytical queries, with ACID transaction semantics.
• Datasets are the top-level container for tables and views.
• Query external data without loading it: external tables and federated queries over Cloud Storage, Bigtable, Spanner, or Google Sheets.
• ANSI-standard SQL, BigQuery ML for models in SQL, BigQuery DataFrames (Python), notebooks in BigQuery Studio, and connections to Looker, Data Studio, Sheets, Tableau, or Power BI.
• Interfaces: the console, the bq command-line tool, client libraries, REST and RPC APIs, and ODBC and JDBC drivers.

BigQuery locations and availability

The location is chosen when you create the dataset and determines where data is stored and processed, as well as the price.

Loading data into BigQuery

# Load a CSV from Cloud Storage with a schema
bq load --source_format=CSV mydataset.mytable gs://mybucket/mydata.csv ./myschema.json

# Wildcards in the Cloud Storage URI
bq load --source_format=CSV --autodetect mydataset.mytable "gs://mybucket/fed-*/temp/*.csv"

Schema auto-detection doesn't detect headers if all columns are strings. Loading a local file also uses bq load.

BigQuery pricing models

Storage is billed separately: the first 10 GiB each month are free, and tables or partitions not modified for 90 consecutive days move to cheaper long-term storage. Cost estimation is covered in domain D3.

An asynchronous, scalable messaging service that decouples services producing messages from services processing them, with typical latencies around 100 milliseconds. Publishers send messages to a topic and subscribers receive them through subscriptions.

• It combines Kafka's horizontal scalability with middleware features such as dead-letter queues and filtering.
• Per-message parallelism, not per partition: the number of concurrent clients isn't limited by partitions.
• It is meant for service-to-service communication, not with mobile or IoT end clients.

Subscription types and delivery

• A subscription belongs to a single topic; a topic can have several. It only receives messages published after it was created.
• The subscriber must acknowledge each message before the ackDeadline; otherwise it is redelivered.
• Default: at-least-once delivery with no ordering. You can enable ordering by ordering key and exactly-once delivery. Subscribers should be idempotent.
• Subscriptions expire after 31 days of inactivity unless you change the expiration policy.
• Topic retention lets you seek back and replay messages already published.

gcloud pubsub topics create orders
gcloud pubsub subscriptions create orders-sub --topic=orders
gcloud pubsub topics publish orders --message="hello"
gcloud pubsub subscriptions pull orders-sub --auto-ack

Batch and streaming data processing with the same programming model. Pipelines read from one or more sources, transform the data, and write it to a destination.

• Fully managed: it allocates a pool of worker VMs for each job and deletes them when it ends. It autoscales and rebalances work automatically.
• Built on Apache Beam (Java, Python, and Go SDKs): the same pipeline can run on Flink or Spark without rewriting it.
• Exactly-once processing by default; for streaming you can use at-least-once mode to cut cost and latency.
• Ways to build pipelines: code with the Beam SDKs, templates (including Google's, no coding required), and JupyterLab notebooks.
• Use cases: moving data, ETL into BigQuery, BI dashboard backends, real-time ML, processing logs and sensor data.
• The console monitoring interface shows the pipeline graph and each stage's progress (job status is reviewed in domain D3).

Managed Service for Apache Kafka

A Google Cloud service to run secure, scalable open-source Apache Kafka clusters.

It doesn't allow unauthenticated or unencrypted connections. Authorization combines IAM (cluster management) and Kafka ACLs.

Pub/Sub or Kafka

Exam scenarios

Common exam traps