Monitoring and logging

Study guide for Associate Cloud Engineer, Unit 4 · Topic 6. This is the topic's lecture in reading form — every slide's teaching, figures and worked examples, in order — followed by the official Google Cloud pages its claims rest on.

What the exam guide asks, quoted. Monitoring and logging. Considerations include:

1. Creating Cloud Monitoring alerts based on resource metrics
2. Creating and ingesting Cloud Monitoring custom metrics (e.g., from applications or logs)
3. Exporting logs to external systems (e.g., on-premises, BigQuery)
4. Configuring log buckets, log analytics, and log routers
5. Viewing and filtering logs in Cloud Logging
6. Viewing specific log message details in Cloud Logging
7. Using cloud diagnostics to research an application issue
8. Viewing Google Cloud status
9. Configuring and deploying Ops Agent
10. Deploying Managed Service for Prometheus
11. Configuring audit logs

Monitoring and logging

Eleven considerations, one pipeline: collect, store, route, read, act

Collect — the Ops Agent, Managed Service for Prometheus, Cloud Audit Logs. Store and route — the Log Router, its sinks and log buckets. Read — the Logs Explorer, its query language, one entry's fields. Act — alerts, custom metrics, diagnostic tools, and Google's own status pages.

This is the largest topic in the exam guide: eleven considerations under one heading, monitoring and logging. Taken in the guide's order they look like a list, but they are one pipeline. Telemetry is collected — by the Ops Agent on Compute Engine, by Managed Service for Prometheus on Kubernetes, and by Cloud Audit Logs for administrative actions. Logs are stored and routed — every project has a Log Router, two system sinks and log buckets, and whatever sinks you add. Logs are read — in the Logs Explorer, with a query language, down to one entry's fields. And the data is acted on — alerts on metrics, custom metrics you define, the tools that find a new error or a slow request, and Google's own status pages when the problem is not yours. The deck follows the guide's order; each slide names the command and the consequence the exam likes to test.

Creating alerts on resource metrics

A policy says when to alert; a channel says who hears about it

• Alerting policy: the conditions that describe when, plus notification channels
• Metric-threshold condition: above or below a threshold for a whole retest window
• Metric-absence condition: no data at all for the retest window
• Build it in the console, the API, gcloud or Terraform; gcloud accepts a policy file

Worked example (synthetic). A web tier's CPU utilization should never sit above its limit for five minutes. The engineer selects the metric, sets a threshold and a retest window, and attaches an email channel and a Slack channel — two different kinds, as Google recommends.

Cloud Monitoring's alerting has three parts, and you configure the first two. An alerting policy describes the circumstances under which you want to be alerted and how you want to be notified. It can watch time-series data stored by Monitoring or logs stored by Cloud Logging, and when the data meets the policy's condition, Monitoring creates an alert and sends the notifications. For resource metrics, two condition types matter most. A metric-threshold condition is met when a metric's values are above or below a threshold for a specific retest window. A metric-absence condition is met when a monitored time series has no data at all for the retest window — the alert for something that has gone silent. A notification channel defines how you receive notifications, and one policy can hold several. In the console you click Select a metric, choose the resource type and metric type, and then configure the trigger. Google recommends several types of channel for redundancy, because the mobile app, PagerDuty, webhooks and Slack all share a single point of failure. Policies can also be created through the Cloud Monitoring application programming interface, the gcloud tool or Terraform. With gcloud monitoring policies create, a policy can be passed as a file with the policy-from-file flag, or a basic policy built from command-line flags.

From a metric to a notification

The condition is judged over a retest window, not a single point

Figure: A flow from a metric time series to a decision: is the condition met for the whole retest window? If not, evaluation continues; if so, Monitoring creates an alert that goes to two notification channels, email and Slack. A dashed path shows a log entry with a particular message reaching the alert through a log-based alerting policy.

Worked example (synthetic). A thirty-second CPU spike under a five-minute retest window opens nothing; the same load held for five minutes opens one alert, delivered to both channels.

Drawn as a flow, the policy's logic is simple and the exam's trap is the window. The condition is evaluated against the time series over a retest window: a metric-threshold condition is met only when the value stays past the threshold for that whole window, so a brief spike does not alert. When the condition is met, Monitoring creates an alert and sends it to every channel on the policy. The dashed path is the other kind of policy: a log-based alerting policy, which notifies you when a particular message appears in your logs rather than when a metric crosses a line.

Creating and ingesting custom metrics

From your code, or from your logs — two different routes

• User-defined (custom) metrics: any metric Google Cloud does not define
• From applications: Google recommends OpenTelemetry over the Monitoring API
• From logs: log-based metrics, which must be created in Cloud Logging
• gcloud logging metrics create --log-filter: a counter or a distribution

Worked example (synthetic). An online shop wants orders per minute, which no built-in metric can know. It instruments its checkout code with OpenTelemetry, and separately counts 'payment declined' log lines with a log-based counter metric.

Built-in metrics cover what Google Cloud can see from outside — backend latency, disk usage — but, in Google's own example, they can't tell you how many background routines your application spawned. For that you create user-defined metrics, sometimes called custom metrics: any metric that isn't defined by Google Cloud, capturing application-specific or client-side data. Once written, Monitoring treats them like built-in metrics. There are two routes, and the exam separates them. From an application, you can call the Cloud Monitoring application programming interface directly — if you write data for a metric that doesn't exist yet, Monitoring creates its descriptor from the data's structure — but Google recommends OpenTelemetry instead. From logs, you create a log-based metric: a Cloud Monitoring metric derived from the content of log entries, which Google says you must create from Cloud Logging. On the command line that is gcloud logging metrics create with a log filter, which counts matching entries or extracts values into a distribution. User-defined log-based metrics are either counters, which count the entries that match a filter, or distributions, which Google says to use for values like latencies; the boolean type, which records whether an entry matches, appears only among Google's system-defined metrics. They are calculated from both included and excluded logs, so an exclusion filter that keeps entries out of storage does not hide them from a metric. One behaviour catches teams out: a new log-based metric is populated only from entries received after it is created — it never backfills.

Two routes to a custom metric

Where you create it decides what it can measure

Worked example (synthetic). A team needs a 'refund issued' count from yesterday onward and has the events in its logs. A log-based counter fits — but its chart will start today, because the metric doesn't backfill.

Side by side, the two routes differ in three ways that decide exam answers. Where it is created: application metrics come from your code, through OpenTelemetry as Google recommends, or through the Monitoring application programming interface, which creates the metric descriptor when you first write data; log-based metrics must be created from Cloud Logging. What it measures: an application metric can carry anything the code knows, while a log-based metric either counts the entries that match a filter or extracts values from them into a distribution. And history: a user-defined log-based metric comes only from entries received after it is created, so it is never populated retroactively. Both kinds behave like any other metric once they exist — alerting policies can monitor them.

Exporting logs to external systems

A sink routes what matches its filter — from the moment it exists

• A sink routes matching log entries to a destination for storage, analytics or streaming
• Destinations: log bucket, BigQuery, Cloud Storage, Pub/Sub, or another project
• Beyond its own project's log buckets, the sink's service account needs write access
• Pub/Sub is Google's recommended path to third-party and on-premises tools

Worked example (synthetic). A retailer routes payment logs to BigQuery to join them with sales data, and security logs to a Pub/Sub topic that its on-premises security tool subscribes to.

Exporting logs means creating a sink. Google describes log sinks as routing log entries from your project to supported destinations for long-term storage, analytics or streaming, and gcloud logging sinks create lists them: a Cloud Logging log bucket, a Cloud Storage bucket, a BigQuery dataset, a Pub/Sub topic, or another Google Cloud project. The destination must already exist, and matching log entries are routed to it after the sink is created — nothing already stored is sent. A sink performs a write, so it must be authorized. For a log bucket in its own project that is automatic; for every other destination the sink is attached to a service account that needs permission to write there — the Storage Object Creator role on a Cloud Storage bucket, or the Pub/Sub Publisher role on a topic. For systems outside Google Cloud, the guide's on-premises example, Google recommends Pub/Sub for integrating Cloud Logging with third-party software: the third party receives the entries by subscribing to the same topic, and Google's Splunk reference architecture notes that Splunk can run on-premises. One timing trap: a new sink to a Cloud Storage bucket might take several hours to start routing.

Which destination, and when

Each destination answers a different need, and needs its own role

Worked example (synthetic). A compliance team must keep seven years of audit logs cheaply and rarely read them. Cloud Storage is the destination — and the sink's service account needs Storage Object Creator on that bucket.

Each destination exists for a reason, and Google states it in one line. A log bucket keeps log data in resources managed by Cloud Logging. A BigQuery dataset is for joining log data with other business data. A Cloud Storage bucket is for long-term storage. A Pub/Sub topic is for exporting log data and using third-party integrations such as Splunk or Datadog. The third column is the one that fails silently when it is missing: except for a log bucket in the sink's own project, the sink's service account must be granted a role on the destination — Logs Bucket Writer on a log bucket elsewhere, BigQuery Data Editor on a dataset, Storage Object Creator on a bucket, Pub/Sub Publisher on a topic.

One sink for a whole folder

Attach the sink above the projects, and new projects are covered too

Figure. An organization resource holding three folders — Production, Development and Sandbox — each containing one project, with a central logging project at the root. An aggregated sink is attached to the Production folder. The callout says a sink on a folder with the include-children option routes every project inside it and no project outside it.

Worked example (synthetic). A security team creates one aggregated sink on the Production folder with --include-children. A project created there next month is routed without anyone touching the sink.

When the requirement is every project, a sink per project is the wrong shape: Google notes that sinks typically route only the log entries that originate in their own resource, so each new project would need another one. An aggregated sink is created on a folder or the organization and routes logs from that resource and its child resources to a centralized location. On the command line it is gcloud logging sinks create with the include-children option, which, in the reference's words, exports logs from all child projects and folders. The figure shows the scope. Attached to the Production folder, the sink reaches every project inside it — including ones created later — and no project outside it. Google's Splunk reference architecture starts exactly here: Cloud Logging collects the logs into an organization-level aggregated sink and sends them to Pub/Sub.

Configuring log buckets, analytics and routers

Every log entry passes through a router, a sink and a bucket

• Every project has a Log Router and two sinks: _Required and _Default
• _Required: a subset of audit logs, 400 days, sink and retention fixed
• _Default and user-defined buckets: 30 days by default, 1 to 3650 configurable
• Upgrade a bucket to Observability Analytics for SQL, then link a BigQuery dataset
• A bucket's region and its analytics upgrade are both permanent

Worked example (synthetic). A platform team creates a user-defined bucket in europe-west1 with 365-day retention and analytics enabled, then points a sink at it — having confirmed the region first, because that choice is final.

Storage and routing live in three objects. Each project, billing account, folder and organization has a Log Router, which manages the flow of log entries through its sinks. Cloud Logging creates two sinks everywhere, named _Required and _Default, and between their filters every log entry that originates in the resource is routed by one of them. The _Required sink routes a subset of audit logs to the _Required bucket; you can't modify or delete that sink, and you can't change that bucket's retention — Google's table lists it at 400 days, not configurable. The _Default sink routes to the _Default bucket, and you can edit it: change its destination, add exclusion filters. Every sink has one inclusion filter and can have several exclusion filters, and an entry that matches an exclusion filter isn't routed. User-defined buckets are created in projects — not in folders or organizations — with gcloud logging buckets create, and then a sink routes logs into them. Their retention defaults to 30 days and, like the _Default bucket's, can be set anywhere from 1 to 3650 days. The guide calls the analysis feature log analytics; Google now calls it Observability Analytics, a structured query language interface you get by upgrading a bucket — the gcloud flag is enable-analytics, and the gcloud reference still describes such a bucket as enrolled into Log Analytics. An upgraded bucket can also get a linked, read-only BigQuery dataset, without ingesting the data into BigQuery. Two choices are permanent: a bucket's region can't be changed after creation — to move, you create a new bucket in the other region, redirect the sinks to it and delete the old one — and an analytics upgrade can't be removed.

One entry, through the Log Router

Two system sinks are always there; your sinks add destinations

Figure: A log entry enters the Log Router, which sends a subset of audit logs through the _Required sink to the _Required bucket, fixed at 400 days; everything else through the _Default sink to the _Default bucket, 30 days by default; and any entry that matches a user sink's inclusion filter and no exclusion filter to that sink's destination — a log bucket, BigQuery, Cloud Storage or Pub/Sub.

Worked example (synthetic). A team adds an exclusion filter for noisy debug entries to the _Default sink. Those entries stop being stored in the _Default bucket, but a separate sink of their own can still route them elsewhere.

Here is one entry's path. The Log Router receives it and hands it to every sink in the resource. The two system sinks between them route every entry that originates there: the _Required sink takes a subset of audit logs to the _Required bucket, where retention is fixed at 400 days, and the _Default sink takes the rest to the _Default bucket, 30 days unless you change it. Your own sinks sit beside them. Each has one inclusion filter and optional exclusion filters, and an entry is routed to the sink's destination when it matches the inclusion filter and no exclusion filter.

The three kinds of log bucket

What lands in each, how long it stays, and what you can change

Worked example (synthetic). An auditor asks for a year of Admin Activity history. The _Required bucket already holds 400 days of it, and no one could have shortened that retention.

The three buckets differ in what you can touch. The _Required bucket holds the audit logs Google always writes — Admin Activity and System Event — for 400 days, and neither its sink nor its retention can be changed. The _Default bucket receives everything else, including Data Access audit logs unless you route them elsewhere; its retention defaults to 30 days and can be set from 1 to 3650 days in a project, and its sink can be edited. A user-defined bucket holds whatever your own sinks send it, with the same retention range. Its two one-way doors are the ones to remember: the region you choose at creation, and an upgrade to Observability Analytics.

Viewing and filtering logs

A query is a Boolean expression, with a precedence rule to memorize

• Logs Explorer: query pane, fields pane, timeline, query results, time-range selector
• Comparisons: resource.type = "gce_instance" AND severity >= "ERROR"
• Boolean operators must be capitalized; lowercase and, or, not are search terms
• NOT binds first, then OR, then AND — so use parentheses
• gcloud logging read returns newest first; --freshness defaults to one day

Worked example (synthetic). An engineer searching for failed logins types and between two conditions and gets far more entries than expected — lowercase and is just another search term. Capitalizing it fixes the query.

Cloud Logging offers two interfaces: the Logs Explorer, for troubleshooting and exploring log data, and Observability Analytics, for joining log and trace data or generating insights and trends. The Logs Explorer is where you retrieve, view and analyze log entries stored in log buckets. It has a query pane, a fields pane, a timeline and a query results pane, with a time-range selector in the toolbar. The query language is the same one used by the Logging application programming interface and the command line, and by the filters of sinks and log-based metrics. A query is a Boolean expression that selects a subset of log entries, built from comparisons of the form field, operator, value — Google's example is resource.type equals gce_instance AND severity greater than or equal to ERROR. A bare value with no field is a search across the entry: each field is compared using the has operator. Three rules catch people. Boolean operators always need to be capitalized; lowercase and, or and not are parsed as search terms. The NOT operator has the highest precedence, followed by OR and AND in that order — OR is evaluated before AND, the opposite of the usual convention — so Google says to nest mixed AND and OR rules in parentheses. And on the command line, gcloud logging read returns entries newest first, but its freshness flag defaults to one day and applies only to filters without a timestamp — a filter with no time bound quietly means the last day.

Query syntax at a glance

Four patterns cover most of what the exam asks

Figure. Four cards. Field comparison: resource.type equals gce_instance AND severity at least ERROR. Search the payload: textPayload colon SyncAddress, where the colon is the has operator. Precedence: a OR NOT b AND NOT c OR d means (a OR (NOT b)) AND ((NOT c) OR d). Command line: gcloud logging read with a resource.type filter returns only the last day unless the freshness flag says otherwise.

Worked example (synthetic). A query written as severity>=ERROR OR resource.type="gce_instance" AND logName:syslog groups the OR first. Adding parentheses makes the intended grouping explicit.

Four patterns cover most questions. A field comparison names a field, an operator and a value, joined with capitalized AND. The has operator, a colon, searches within a field — Google's gcloud example reads entries whose payload includes the word SyncAddress. Precedence is the third card, and it is Google's own example: a OR NOT b AND NOT c OR d is evaluated as a OR not-b, AND, not-c OR d — the ORs group first. The fourth card is the command line: gcloud logging read with a filter such as resource.type equals gce_instance, which, without a timestamp in the filter, returns only the last day unless you set the freshness flag.

Viewing a specific log entry's details

Expand an entry: when it happened, what produced it, and the payload

Figure. A two-column field guide to a log entry: resource, the monitored resource that produced it; timestamp, when the event occurred; receiveTimestamp, when Logging received it; severity, from DEFAULT to EMERGENCY; the payload in textPayload, jsonPayload or protoPayload; httpRequest and trace. Beneath: clicking a value offers Show matching entries or Hide matching entries.

Worked example (synthetic). An engineer expands one ERROR entry from a virtual machine, clicks its serviceName value and chooses Show matching entries — the query is rewritten to every entry with that value.

Viewing a specific log message means expanding it in the query results pane, and knowing what each field means. The resource field is the monitored resource that produced the entry. There are two times: timestamp is when the event described by the entry occurred, and receiveTimestamp is when Logging received it, so the two can differ. Severity places the entry on Google's scale, from DEFAULT, meaning no severity assigned, up to EMERGENCY. The payload sits in one of three fields: textPayload for a Unicode string, jsonPayload for a structure expressed as a JavaScript Object Notation, or JSON, object, and protoPayload for a protocol buffer — the field audit log entries use, since the audit log type is one of its supported types. When an entry came from a web request, httpRequest describes that request, and trace carries the identifier written to Cloud Trace. The Logs Explorer also turns any field into a query: click a field's value, such as a serviceName, and choose Show matching entries or Hide matching entries.

Severity levels, low to high

severity >= ERROR means ERROR and every level above it

Worked example (synthetic). A filter of severity >= ERROR returns ERROR, CRITICAL, ALERT and EMERGENCY entries; WARNING entries, at 400, stay out.

Severity is a ranked number, which is what makes comparisons like greater-than-or-equal work. The table shows selected levels with their values: DEFAULT is zero, meaning no severity was assigned; WARNING, at 400, might cause problems; ERROR, at 500, is likely to; CRITICAL, at 600, causes more severe problems or outages; ALERT, at 700, means a person must take action immediately; and EMERGENCY, at 800, means one or more systems are unusable. So severity greater than or equal to ERROR returns ERROR and everything above it.

Using cloud diagnostics to research an issue

Match the symptom to the tool: errors, latency or resource use

• Error Reporting groups error events by root cause, automatically
• Cloud Trace follows one request across services and shows its latency
• Cloud Profiler samples CPU and memory in production at low overhead
• Error Reporting needs the log entries stored in log buckets

Worked example (synthetic). After a release, checkout feels slow but nothing errors. Error Reporting stays quiet; Cloud Trace shows every slow request waiting on one downstream call, and Profiler then shows where that service spends its CPU.

The guide's phrase cloud diagnostics names no single product. Google groups this work under application performance monitoring, or APM, which it describes as monitoring, diagnosing and managing the performance, availability and user experience of applications, and three tools do the researching. Error Reporting aggregates and groups application error events by their root cause. It is enabled automatically, and it infers errors by scanning log entries for stack traces and common error patterns — which is why Google notes that using Error Reporting requires your log entries to be stored in log buckets. Cloud Trace is a distributed tracing system that tracks request latency across services: a trace is the path of one request across the components of a distributed application, so it answers questions like why some requests take longer than others. Cloud Profiler is a statistical, low-overhead profiler that continuously gathers central processing unit usage and memory-allocation information from production applications. The exam gives you a symptom: errors point to Error Reporting, a slow request to Trace, a resource-hungry service to Profiler.

Which tool answers which question

Start from the symptom the scenario describes

Worked example (synthetic). An item says latency doubled for one request path that crosses four services, with no errors. Error Reporting has nothing to group; Trace is the answer.

Read the scenario for its symptom and the tool follows. New or frequent exceptions are Error Reporting's job, because it groups error events by their root cause. Slow requests that cross services are Cloud Trace's, because a trace follows one request across components with its latency. A service that burns processor time or memory in production is Cloud Profiler's, because it continuously samples central processing unit and memory-allocation data at low overhead. And when you need the exact record of what happened, the Logs Explorer retrieves and filters the log entries themselves.

Viewing Google Cloud status

Is it us, or is it Google? Two places answer that

Worked example (synthetic). During a regional incident, the on-call engineer opens Personalized Service Health and checks whether the incident lists the products and locations their project uses, before paging anyone.

When something breaks, the first diagnostic question is whether the cause is yours or Google's, and the guide calls this viewing Google Cloud status. There are two places. Personalized Service Health is Google's primary channel for incident information relevant to your projects. Its dashboard in the console shows incidents that affect your project, their state, and the impacted products and locations; the Service Health application programming interface pulls the same events per project or organization; and alerts are available because the events are logged in Cloud Logging. Google Cloud Service Health is the public page — until March 2022 it was called the Google Cloud Status Dashboard. It covers ongoing widespread incidents that meet certain criteria, through a public dashboard, a public feed and the console, and Google positions it as the fallback for when Personalized Service Health is unavailable; Personalized Service Health always has the most information. For a question about your own projects, the answer is Personalized Service Health.

Configuring and deploying the Ops Agent

One agent for logs, metrics and traces, configured by override

• Collects logs, metrics and traces on Compute Engine: Fluent Bit plus OpenTelemetry
• Install: add-google-cloud-ops-agent-repo.sh --also-install; no legacy agents
• Fleets: agent policies, with gcloud compute instances ops-agents policies
• Override in /etc/google-cloud-ops-agent/config.yaml, then restart the agent

Worked example (synthetic). An engineer needs one application's log file collected on fifty machines. An agent policy installs the Ops Agent across the fleet, a config.yaml override adds the file's path, and after a restart the new logs appear.

The Ops Agent is Google's primary agent for collecting telemetry from Compute Engine instances. It combines logs, metrics and traces in one process, using Fluent Bit for logs and the OpenTelemetry Collector for metrics and traces, and sends logs to Cloud Logging and metrics to Cloud Monitoring. On one virtual machine, the repository script run with the also-install flag installs the agent, which then starts automatically. Two prerequisites: the machine must not already have the legacy Logging or Monitoring agent, and it needs credentials to write — by default the Compute Engine service account holds Logs Writer and Monitoring Metric Writer. For a fleet, Google offers agent policies, created with the gcloud compute instances ops-agents policies command group, which use the VM Manager tools to install on new and existing machines. Configuration works by override. The agent has a built-in default configuration — file-based syslog logs and host metrics — that you can't modify directly. You write overrides in /etc/google-cloud-ops-agent/config.yaml on Linux, built from receivers, which say what is collected; processors, which modify it; and a service element that links them into pipelines. The file is merged with the built-in configuration when the agent restarts, so after any change you must restart it. And the agent can't be configured to export to other services — routing is Cloud Logging's job.

How the Ops Agent's configuration is built

Built-in plus your overrides, merged when the agent restarts

Figure: The built-in configuration, collecting syslog logs and host metrics, and your overrides in config.yaml both flow into a merge step that happens when the agent restarts. The merged configuration runs receivers, which say what is collected, then processors, which modify it, then the service pipelines, which send logs to Cloud Logging and metrics to Cloud Monitoring.

Worked example (synthetic). An override that adds a receiver for an application log file changes nothing until the agent restarts; after the restart, the merged pipeline starts sending that file to Cloud Logging.

The configuration model explains the exam's favourite Ops Agent question. There are two inputs: the built-in configuration, which collects syslog logs and host metrics and which you can't edit, and your override file. They are merged when the agent restarts, and not before. The merged configuration is a set of pipelines: receivers say what is collected, processors say how it is modified, and the service element links them together, sending logs to Cloud Logging and metrics to Cloud Monitoring.

Deploying Managed Service for Prometheus

Keep PromQL and Grafana; hand the Prometheus servers to Google

Worked example (synthetic). A team with existing Grafana dashboards moves its Kubernetes workloads to managed collection. The dashboards keep working because they query the same PromQL data, now kept in Monarch for 24 months.

Managed Service for Prometheus is Google Cloud's fully managed, multi-cloud, cross-project solution for Prometheus metrics. It collects from Prometheus exporters and lets you query globally with the Prometheus query language, PromQL, so existing Grafana dashboards and PromQL alerts keep working; the data lives in Monarch, the datastore behind Google's own monitoring, for 24 months. There are four collection modes: managed collection, self-deployed collection, the OpenTelemetry Collector, and the Ops Agent. Google recommends managed collection, because it removes deploying, scaling, sharding and maintaining Prometheus servers. On Google Kubernetes Engine it is enabled by default on Autopilot clusters from version 1.25 and on Standard clusters from 1.27, and you tell it what to scrape with PodMonitoring custom resources. Self-deployed collection is a drop-in replacement for the upstream Prometheus binary, for teams that want to keep running Prometheus themselves. Either way, collectors push data to Google Cloud; Google never reaches into your cluster to pull metrics.

PodMonitoring, one namespace at a time

A PodMonitoring resource sees only the namespace it lives in

Figure. Two cards and a band. Namespace shop has a PodMonitoring resource, so its pods are scraped. Namespace payments has none, so nothing there is scraped. Beneath: deploy the same PodMonitoring resource in each namespace you want scraped.

Worked example (synthetic). Metrics from the payments pods never appear although managed collection is on. Nothing is broken: the only PodMonitoring resource lives in the shop namespace.

One detail of managed collection deserves its own slide, because it produces a failure that looks like a bug. Managed collection uses PodMonitoring custom resources to decide what to scrape, and a PodMonitoring resource scrapes targets only in the namespace it is deployed in. So a resource in the shop namespace collects the shop pods' metrics and nothing else, and the payments namespace stays silent until it has one too. Google's instruction is plain: to scrape targets in multiple namespaces, deploy the same PodMonitoring resource in each namespace.

Configuring audit logs

Two are always on, one is off by default, one is on but excludable

• Admin Activity and System Event: always written, into the _Required bucket
• Data Access: off by default except BigQuery; enable it explicitly, at a cost
• Policy Denied: on by default; can't be disabled, but can be excluded
• Enable Data Access on IAM > Audit Logs, or edit the policy's auditConfigs

Worked example (synthetic). An auditor wants to know who reads a sensitive bucket. The team enables DATA_READ logs for Cloud Storage on the Audit Logs page — reads are recorded from then on, and earlier reads never were.

Cloud Audit Logs answer who did what, where, and when, and there are four types, each with its own default. Admin Activity audit logs record user-driven calls that modify the configuration or metadata of resources; System Event audit logs record Google Cloud systems making such changes. Both are always written — you can't configure, exclude or disable them — and both are stored in the _Required bucket of the project where they were generated. Data Access audit logs record calls that read configuration or metadata, and calls that create, modify or read user-provided data. Except for BigQuery, they are disabled by default because they can generate large volumes of data, so for any other service you must enable them explicitly, and doing so can add log charges; they land in the _Default bucket, and reading them there needs more than the Logs Viewer role. Policy Denied audit logs record access denied by a security policy; they are generated by default, and you can't disable them, though exclusion filters can keep them out of storage. You configure Data Access logs on the Audit Logs page of the identity and access management console, or through the policy's auditConfigs section: read the project's policy into a file, change only the auditConfigs section, and write it back with gcloud projects set-iam-policy. Three details: the permission types are ADMIN_READ, DATA_READ and DATA_WRITE; you can exempt specific principals; and you can't disable at a project a Data Access log that a parent folder or organization enabled.

The four audit log types

Who writes each one, and whether you can turn it off

Worked example (synthetic). A scenario asks which audit log shows that an autoscaler added a VM to a managed instance group at 3 a.m. No user did it, so it is System Event, and it was written without any configuration.

Side by side, the four types sort by who writes them and whether you control them. Admin Activity is written when users change configuration or metadata, and System Event when Google Cloud systems do — such as an autoscaler adding a machine to a managed instance group. Both are always written, can't be turned off, and land in the _Required bucket. Data Access is the one you control: off by default except for BigQuery, enabled per service, with principals you can exempt. Policy Denied is on by default; it can't be disabled, but exclusion filters can keep it out of storage.

What this topic actually tests

Five traps hidden behind eleven considerations

New metrics and sinks never backfill — only entries from their creation on. Two things are permanent — a bucket's region and its analytics upgrade. OR binds before AND — and lowercase operators are search terms. The Ops Agent reads its file on restart. Data Access audit logs are off — except BigQuery's.

Close on the five traps this topic hides. First, nothing backfills: a new log-based metric counts only entries received after it is created, and a new sink routes only entries that arrive after it exists. Second, two choices are permanent: a log bucket's region, and its upgrade to Observability Analytics. Third, the query language has its own precedence — NOT, then OR, then AND — and lowercase operators are just search terms. Fourth, the Ops Agent reads its override file only when it restarts. Fifth, Data Access audit logs are off by default for every service except BigQuery, so reads that happened before you enabled them were never recorded. Unit five turns to identity and access management, where the audit logs you just met become the evidence.

Official sources for this topic

• Associate Cloud Engineer exam guide — Section 4
• Alerting overview — Cloud Monitoring
• Create metric-threshold alerting policies — Cloud Monitoring
• Create and manage notification channels — Cloud Monitoring
• gcloud monitoring policies create — reference
• User-defined metrics overview — Cloud Monitoring
• Create user-defined metrics with the API — Cloud Monitoring
• Log-based metrics overview — Cloud Logging
• gcloud logging metrics create — reference
• Route logs to supported destinations — Cloud Logging
• gcloud logging sinks create — reference
• View logs routed to Pub/Sub — Cloud Logging
• Stream logs from Google Cloud to Splunk — Architecture Center
• Routing and storage overview — Cloud Logging
• Configure log buckets — Cloud Logging
• Quotas and limits — Cloud Logging
• gcloud logging buckets create — reference
• Analyze logs using Logs Explorer and Observability Analytics — Cloud Logging
• Logs Explorer interface — Cloud Logging
• Logging query language — Cloud Logging
• gcloud logging read — reference
• LogEntry — Cloud Logging API reference
• Observability in Google Cloud
• Error Reporting overview
• Cloud Trace overview
• Cloud Profiler overview
• Personalized Service Health overview
• Use Google Cloud Service Health as a fallback
• Ops Agent overview
• Install the Ops Agent
• Agent policies for the Ops Agent
• Configure the Ops Agent
• Google Cloud Managed Service for Prometheus
• Get started with managed collection — Managed Service for Prometheus
• Cloud Audit Logs overview
• Enable Data Access audit logs — Cloud Audit Logs
• Collate and route organization- and folder-level logs — Cloud Logging