Topic 9 of 10
Cloud Logging and audit logs
The second part of section 3.4 covers logs: configuring audit logs, exporting them with sinks, storing them in buckets, querying them, and enabling network logs. Log-based metrics are in the Cloud Monitoring article.
Contents
Cloud Audit Logs
Every project, folder, organization, and billing account has four types of audit logs. Their entries are immutable.
| Type | What it records | Can you disable it? |
|---|---|---|
| Admin Activity | Calls that change configuration or metadata: creating a VM, changing IAM permissions | No. Written even if you disable the Cloud Logging API |
| Data Access | Reads of configuration or metadata, and calls that create, modify, or read user data | Disabled by default (except BigQuery); you must enable them |
| System Event | Changes made by Google systems, not users; for example, MIG autoscaling | No |
| Policy Denied | Access denied because of a security policy | No, but you can use exclusion filters so they aren't stored |
# Read the project's audit logs gcloud logging read "logName : projects/PROJECT_ID/logs/cloudaudit.googleapis.com" \ --project=PROJECT_ID --freshness=7d # Log names projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fdata_access projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fpolicy
Public resources (allUsers or allAuthenticatedUsers) don't generate audit logs, to protect end-user identities. By default gcloud logging read only reads the last day; --freshness extends the period.
Enabling Data Access audit logs
You enable them per service on the IAM Audit Logs page in the console or in the auditConfigs section of the project, folder, or organization IAM policy. They can generate a lot of volume and are chargeable. Google recommends enabling them when possible because they help support.
ADMIN_READ
Reads of metadata or configuration.
DATA_READ
Reads of user data.
DATA_WRITE
Writes of user data.
If you enable them for Cloud Storage, authenticated browser downloads from storage.cloud.google.com can return 403, because the log can't be generated with cookie-based authentication.
Log Router and sinks
Every project, folder, organization, and billing account has a Log Router that passes each entry through the resource's sinks. Each sink evaluates the entry independently with its inclusion filter and exclusion filters, so several sinks can route the same entry.
_Required sink
Sends Admin Activity and System Event logs to the _Required bucket. It can't be modified or deleted.
_Default sink
Sends everything else to the _Default bucket. You can edit it, change its destination, add exclusions, or disable it.
# Project sink to a Cloud Storage bucket gcloud logging sinks create archive-sink storage.googleapis.com/my-log-archive \ --log-filter='severity>=WARNING' # Aggregated folder sink (includes child projects) gcloud logging sinks create folder-sink \ bigquery.googleapis.com/projects/central-logs/datasets/all_logs \ --folder=FOLDER_ID --include-children --log-filter='logName:"cloudaudit.googleapis.com"'
- Not retroactive: a sink only routes entries received after it's created. You can, however, copy logs already stored in a log bucket to Cloud Storage.
- Aggregated sinks on folders or organizations: non-intercepting ones don't change routing for children; intercepting ones (
--intercept-children) stop the entry from reaching child sinks, except the _Required sink of the originating resource. - Each sink has a writer identity (a service account) that needs permission on the destination, unless it writes to a log bucket in the same project.
- A misconfigured sink routes nothing; error logs are written and an email goes to the Essential Contacts.
Sink destinations
| Destination | What for | Writer identity role |
|---|---|---|
| Log bucket | Store in Cloud Logging and use Logs Explorer, Observability Analytics, and Error Reporting | roles/logging.bucketWriter |
| Cloud Storage | Long-term storage, as JSON files | roles/storage.objectCreator |
| BigQuery | Join logs with other business data | roles/bigquery.dataEditor |
| Pub/Sub | Export outside Google Cloud (Splunk, Datadog…) | roles/pubsub.publisher |
| Another project | The destination project's sinks reroute the entries | roles/logging.logWriter |
Google also says to grant roles/logging.logWriter for every destination. If you delete a sink's destination, delete the sink too.
Log buckets and retention
| Bucket | Retention |
|---|---|
| _Required | 400 days, not configurable |
| _Default (project) | 30 days by default, configurable from 1 to 3650 days |
| _Default (folder or organization) | 30 days, not configurable |
| User-defined | 30 days by default, configurable from 1 to 3650 days |
gcloud logging buckets create audit-bucket --location=global \ --enable-analytics --retention-days=365
- Keeping logs longer than the default retention is chargeable.
- You can't change a bucket's region after creating it.
- You can lock retention for compliance: afterwards the bucket can't be deleted until every entry has met its retention.
- If you route Admin Activity logs to another project, its system sinks exclude them: to store them there, edit the _Default sink or create your own.
Logs Explorer, queries, and Observability Analytics
Logs Explorer
For troubleshooting and exploration. Uses the Logging query language (the same as the API and gcloud). Shows an error histogram. No aggregate operations such as counting entries by pattern.
Observability Analytics
SQL queries that group and aggregate logs, with tables and charts, and join logs with traces. Requires analytics enabled on the bucket. With a linked BigQuery dataset (read-only) you query logs from BigQuery without exporting them.
resource.type="gce_instance" severity>=ERROR jsonPayload.message:"timeout" NOT logName:"cloudaudit.googleapis.com" jsonPayload.message =~ "foo.*bar"
- The Boolean operators
AND,OR, andNOTmust be uppercase; lowercase ones are treated as search terms. Multiple lines are joined with AND. :matches a substring;=~and!~match regular expressions.- Always-indexed fields (faster queries):
resource.type,resource.labels.*,logName,severity,timestamp,httpRequest.status,labels.*…
VPC Flow Logs and firewall logs
VPC Flow Logs
- Samples packets from VMs (including GKE nodes), Cloud Run or App Engine with Direct VPC egress, VLAN attachments, and VPN tunnels, and aggregates them per connection. It doesn't capture every packet.
- For network monitoring, forensics, security analysis, and network cost optimization.
- Enabled at the organization, VPC network, subnet, VLAN attachment, or VPN tunnel level.
- 5-second aggregation interval by default; configurable secondary sampling rate and filters.
- Kept 30 days in Logging by default.
Firewall rules logging
- Enabled rule by rule, whether allow or deny, ingress or egress.
- Creates one record per connection (not per packet) with IPs, protocol, ports, and the rule applied.
- TCP and UDP only. Not supported for the implied deny-ingress and allow-egress rules.
- Useful to check that a deny rule works or how many connections it affects.
gcloud compute networks subnets update subnet-eu --region=europe-west1 \ --enable-flow-logs --logging-aggregation-interval=interval-30-sec \ --logging-flow-sampling=0.5
Cloud Logging roles
| Role | What it allows |
|---|---|
Logs Viewerroles/logging.viewer | View logs, including Admin Activity, System Event, and Policy Denied logs. Not Data Access logs in the _Default bucket |
Private Logs Viewerroles/logging.privateLogViewer | The above plus private logs, such as Data Access logs |
Logs Writerroles/logging.logWriter | Write entries (used by the Ops Agent and writer identities) |
Logs Configuration Writerroles/logging.configWriter | Create and edit sinks, buckets, exclusions, and log-based metrics |
Logging Adminroles/logging.admin | All permissions for every Cloud Logging feature |
The legacy basic Editor role (roles/editor) doesn't include permission to view Data Access logs.
Exam scenarios
| Situation | Answer | Why |
|---|---|---|
| Find out who deleted a VM last week | Admin Activity audit logs | Always on; kept 400 days in _Required. |
| Record who reads objects in a Cloud Storage bucket | Enable Data Access audit logs (DATA_READ) | Disabled by default except for BigQuery. |
| Keep logs for 7 years at the lowest cost | Sink to a Cloud Storage bucket | Long-term storage as JSON files. |
| Centralize logs from every project in a folder | Aggregated sink with --include-children | Routes logs from the folder and its children. |
| Send logs to Splunk | Sink to a Pub/Sub topic | Destination for third-party integrations. |
| Join logs with business data in SQL | Sink to BigQuery, or a bucket with Observability Analytics and a linked dataset | The linked dataset is read-only. |
| A developer needs to read Data Access logs | roles/logging.privateLogViewer | Logs Viewer can't see them; neither can Editor. |
| Analyze which IPs talked to a compromised VM | VPC Flow Logs on the subnet | Sampled and aggregated per connection. |
Common exam traps
❓ Can you disable Admin Activity audit logs to save money?
No. They are always written and stored in _Required, even if you disable the Cloud Logging API.
❓ Are Data Access audit logs on by default?
No, except for BigQuery. You enable them per service and they are chargeable.
❓ Does a new sink export last week's logs?
No. It only routes entries that arrive after it was created.
❓ Can you change the retention of the _Required bucket?
No. It's a fixed 400 days. A project's _Default bucket retention is configurable.
❓ Can Logs Explorer count entries grouped by hour?
It doesn't support aggregations. Use Observability Analytics with SQL.
❓ Does VPC Flow Logs record every packet?
No. It samples packets and estimates total traffic.
Did you get this topic?
Test what you just learned
Your security team must keep audit logs from every project in the organization for 5 years in one central project and query them with SQL. What do you configure?
Ready to test "Cloud Logging and audit logs"?
Original questions, each with a full explanation.
Start a mock exam