Operational excellence and reliability at scale

Study guide for Cloud Digital Leader, Unit 6 · Topic 2. This is the topic's lecture in reading form — every slide's teaching, figures and worked examples, in order — followed by the official Google Cloud pages its claims rest on.

What the exam guide asks. Discuss the fundamental concepts of modern operations, reliability, and resilience in the cloud.

Objectives, quoted from the exam guide:

1. Describe the benefits of modernizing operations by using Google Cloud.
2. Define important cloud operations terms.
3. Describe the importance of designing resilient, fault-tolerant, and scalable infrastructure and processes for high availability and disaster recovery.
4. Define key cloud reliability, DevOps, and SRE terms.
5. Describe how organizations benefit from using Google Cloud Customer Care to support their cloud adoption.
6. Describe the life of a support case during the Google Cloud Customer Care process.

Operational excellence and reliability at scale

See it, survive it, measure it, and get help with it

See it — modern operations and the words for what they collect. Survive it — resilient design, high availability and disaster recovery. Measure it — the terms DevOps and site reliability engineering use for reliability. Get help with it — Customer Care, and what happens to a support case.

The guide summarizes this topic as the fundamental concepts of modern operations, reliability and resilience in the cloud, and its six objectives come in three pairs. The first pair is about seeing a system: what modernizing operations with Google Cloud buys an organization, and the vocabulary those operations use. The second pair is about surviving failure and measuring how well you do: why resilient, fault-tolerant design matters for high availability and disaster recovery, and the reliability terms that DevOps and site reliability engineering teams live by. The third pair is about what happens when something still goes wrong and you need Google: the value of Google Cloud Customer Care, and the life of a support case from the moment it is opened to the moment it is closed. Unit 1 Topic 2 already taught zones and regions as failure domains, so this deck builds on that rather than repeating it.

What modernizing operations buys

From reacting to failures to seeing them coming

• Telemetry is collected automatically for most Google Cloud services
• Understanding behavior lets teams anticipate and respond quickly
• Automation, orchestration and data-driven insight replace toil
• A cultural shift toward continuous learning and experimentation

Worked example (synthetic). A retailer's operations team learned about outages from customer complaints. After moving to Google Cloud, an alerting policy pages them when checkout latency climbs — before the complaints start.

The first objective asks for the benefits of modernizing operations with Google Cloud, and Google's framework states the goal in one line: operational excellence in the cloud involves designing, implementing and managing cloud solutions that provide value, performance, security and reliability. Three things change when operations modernize. First, you can see the system without building the instruments yourself. Cloud Monitoring automatically collects and stores performance information for most Google Cloud services, and Google Cloud Observability exists to help you understand the behavior, health and performance of your applications. The payoff is stated plainly: understanding how applications behave and how components connect helps you anticipate, identify and respond to unexpected changes quickly and effectively. Second, the repetitive work shrinks. To achieve operational excellence, Google says you should embrace automation, orchestration and data-driven insights, because automation helps to eliminate toil — the manual, repetitive work that grows with the system. Third, and easiest to miss, it is not only technical. Google's framework says operational excellence includes a cultural shift that encourages continuous learning and experimentation. An organization that buys the tools and keeps the old habits has modernized its dashboards, not its operations.

Operations before and after

The same incident, run two ways

Figure. Two columns comparing traditional and modernized operations across the same four steps of an incident: how it is noticed, how it is investigated, how the fix is repeated, and how it is reviewed afterwards.

Worked example (synthetic). An exam item asks which benefit comes from telemetry being collected automatically. The graded answer is faster detection and diagnosis — not a smaller bill.

Put an incident through both columns and the benefit becomes concrete. In the traditional column, the team hears from a customer, logs in to servers to look around, fixes the problem by hand, repeats that fix by hand the next time, and asks who made the mistake. In the modernized column, each step has a Google mechanism behind it. Detection: to be notified when the value of a performance metric meets criteria you define, you create an alerting policy in Cloud Monitoring. Diagnosis: the telemetry is already there, because Cloud Monitoring collects performance information for most services automatically. Repetition: automation helps to eliminate toil. And review: Google says a successful post-incident review needs a blameless culture that focuses on learning and improvement rather than assigning blame. Every row is the same incident; what changed is how early the team knows, how fast it can see, and whether the next one is less likely.

Cloud operations terms

Observability, telemetry, and the three kinds of data

Worked example (synthetic). Checkout is slow for some users. The metric says latency rose, the logs say which errors occurred, and the trace says which service in the chain spent the time.

The second objective asks for important cloud operations terms, and Google's observability documentation gives them a clean structure. Observability is a comprehensive approach to collecting and analyzing telemetry data to help you understand the state of your applications and their operating environment. Telemetry, in turn, is three kinds of data. Metric data is numeric data about health or performance that the system measures at regular intervals — for example, central processing unit, or CPU, utilization and request latency. Log data is different: a log is a generated record of system or application activity over time. And trace data is different again: a trace represents the path of a request across the components of your distributed application. Each has a home. Cloud Monitoring holds the metrics and the alerting policies that notify you when a metric meets criteria you define. Cloud Logging is a real-time log-management system with storage, search, analysis and monitoring support. And Cloud Trace is a distributed tracing system that tracks request latency and helps you troubleshoot performance bottlenecks across services. The exam likes to hand you a symptom and ask which kind of data answers it, so learn them by what they are shaped like: a number over time, an event record, or a path.

Which question each kind of telemetry answers

Match the symptom to the data, then to the service

Figure: A flow chart pairing three operational questions with the telemetry that answers each: whether a service is performing well goes to metric data in Cloud Monitoring, what happened and when goes to log data in Cloud Logging, and why a request takes long goes to trace data in Cloud Trace. All three feed observability.

Worked example (synthetic). An item says one API call in ten is slow while averages look fine. The graded answer is Cloud Trace — only a trace follows that one request through each service.

Google's own pages phrase the three services as the questions they answer, and those questions are the best way to remember them. Cloud Monitoring's overview lists questions like: what is the load on my service, is my website responding correctly, is my service performing well. Those are metric questions — a number, measured over time. Cloud Trace's overview asks different ones: why does a request take a long time to complete, and why do some requests take longer than others. Those are path questions, about one request moving through many components. Logs answer the third kind: what exactly happened, in what order — and Cloud Logging automatically collects log data from Google Cloud resources, so the record is there when you need it. Observability is not a fourth thing; it is having all three and analyzing them together.

Designing to survive failure

Resilient, fault-tolerant, scalable — because downtime costs

• Reliability: consistently performing intended functions without interruption
• Resilience: withstanding and recovering from failures while still performing
• Redundancy, fault-tolerant design, monitoring and automated recovery
• High availability through redundancy; recovery planned before it is needed

Worked example (synthetic). A booking platform runs one database in one zone. Its outage last month lost a day of sales. Redundancy across zones would have kept it serving; a tested recovery plan would have bounded the loss.

The third objective asks why resilient, fault-tolerant and scalable infrastructure and processes matter for high availability and disaster recovery. Start with Google's two definitions. Reliability is a system's ability to consistently perform its intended functions within the defined conditions and maintain uninterrupted service. Resilience is part of reliability: the system's ability to withstand and recover from failures or unexpected disruptions, while maintaining performance. The importance is stated as business cost: downtime can lead to lost revenue, decreased productivity and damage to reputation, and resilient systems implement backups, redundancy and replication to keep data accurate and accessible. The practices follow: Google lists redundancy, fault-tolerant design, monitoring and automated recovery processes. Its reliability principles name how to get there — build highly available systems through resource redundancy, take advantage of horizontal scalability, and design for graceful degradation, so a system under strain does less rather than nothing. You met the building blocks in Unit 1 Topic 2: zones and regions are separate failure domains. Disaster recovery goes further than infrastructure. Google's guidance follows one key principle — plan for failure — and it notes that disaster recovery covers much more than infrastructure failures, including software bugs and data corruption, so you need a comprehensive end-to-end plan.

Disaster recovery across two regions

The failover path is idle until it is the only path

Figure. Users reach health-checked routing, which sends traffic to an application and its data in a primary region while it is healthy. Data is replicated to a standby application's data in a recovery region. A dashed failover path, used only on failure, runs from the routing to the recovery region. The callout notes that the recovery time objective is how long the switch takes and the recovery point objective is how much data the replication lag can lose.

Worked example (synthetic). A payments firm's replication runs every fifteen minutes and failover is manual and takes two hours. Its RPO is set by the first number, its RTO by the second — improving one does nothing for the other.

Disaster recovery is the one part of this topic that is a genuine topology, so here it is drawn. Users reach an entry point that checks health and, while the primary region is healthy, sends traffic there. Data is replicated to a recovery region, where a standby copy of the application waits. The dashed path is the failover — idle until the primary fails. The two numbers every disaster recovery plan is written around belong to two different arrows. Google defines a recovery time objective, or RTO, as the maximum acceptable length of time that your application can be offline; that is how long the failover path takes to engage. A recovery point objective, or RPO, is the maximum acceptable length of time during which data might be lost due to a major incident; that is set by how far replication lags behind. Reading them off different arrows explains why fixing one does nothing for the other. And there is a price: typically, the smaller your RTO and RPO values, the more your application will cost to run. Google Cloud features like automated backups and disaster recovery solutions can help you improve your system's resilience — but the targets are a business decision first.

Reliability, DevOps and SRE terms

Indicator, objective, agreement — and the budget between them

• SLI: a measurement of performance, such as good responses over all responses
• SLO: a target value for an SLI over a period — deliberately below 100%
• Error budget: how much the service may fail and still meet the SLO
• SLA: the whole agreement with a customer, penalties included

Worked example (synthetic). A team's SLO is that 99.9% of requests succeed each month. Halfway through the month the error budget is nearly spent, so the team pauses risky releases until reliability recovers.

The fourth objective asks for key reliability, DevOps and site reliability engineering terms, and three of them nest inside one another. A service-level indicator, or SLI, is a measurement of performance; Google's example is availability, the ratio of successful responses to all responses. A service-level objective, or SLO, is a target value for an SLI, measured over a period of time. And a service-level agreement, or SLA, is bigger than either: Google's planning guide calls it the entire agreement that specifies what service is provided, how it is supported, and the performance, penalties and responsibilities of the parties — with SLOs as its specific, measurable characteristics. Google's SRE fundamentals post adds the business edge: an SLA normally involves a promise that, if the SLO is missed, some kind of penalty will be paid. Between the SLO and perfection sits the error budget. A useful SLO targets less than one hundred percent, because the SLO determines the error budget, and the error budget quantifies how much a service can fail during the period and still meet the SLO. That turns reliability into a decision rule: if the error budget is close to depleted, risky actions like pushing new updates might cause you to violate the SLO. A service required to meet one hundred percent has no error budget at all — and so no room to change anything.

Three nested terms and a budget

Each term is built from the one inside it

Figure. Three nested boxes: the SLA on the outside as the whole agreement, the SLO inside it as a target below one hundred percent that is usually tighter internally than in the SLA, and the SLI innermost as the measurement. Beneath, the error budget is described as the room between the SLO and one hundred percent.

Worked example (synthetic). A provider promises customers 99.9% in its SLA but runs to an internal SLO of 99.95%. The gap is deliberate: it warns the team before the contract is at risk.

Drawn as boxes, the three terms show their order. The service-level indicator, the SLI, is innermost — it is only a measurement. The service-level objective, the SLO, wraps it with a target and a time period. The service-level agreement, the SLA, wraps that with everything a contract carries. One detail the exam likes is which of the two targets is stricter. Google's site reliability engineering, or SRE, fundamentals post says that because availability should not be much better than the SLO, the availability SLO in the SLA is normally a looser objective than the internal availability SLO. So the team aims higher than it promises, and the difference is an early warning. The error budget sits alongside the SLO: it is the room between the target and perfection, and it is what makes risk a managed quantity rather than a matter of opinion.

DevOps and SRE in practice

Two traditions, one set of habits

Worked example (synthetic). After an outage, one team names the engineer who pushed the change; another asks what let the change through. Google's framework asks for the second — a blameless post-incident review.

The rest of the vocabulary belongs to two traditions that Google's framework treats side by side. For DevOps teams, it offers guidance on automation, continuous integration and continuous delivery pipelines, and change management, to enable faster and more reliable software delivery. The DevOps Research and Assessment team, known as DORA, has identified and validated a set of capabilities that drive higher software delivery and organizational performance; continuous delivery is one of them, described as making deployment a reliable, low-risk process that can be performed on demand at any time. For site reliability engineers, the framework points to best practices that achieve high levels of service reliability, including monitoring, incident response and automation. Two terms are shared by both. Toil is the repetitive manual work that automation helps to eliminate. And a postmortem is what Google's framework calls a post-incident review: after an incident, a detailed review to identify the root cause, contributing factors and lessons learned — which only works, Google says, in a blameless culture that focuses on learning and improvement rather than assigning blame.

Google Cloud Customer Care

Four support services, matched to how much is at stake

Worked example (synthetic). A bank running payments on Google Cloud needs someone who already knows its environment when an incident starts. Premium Support is the service that assigns a Technical Account Manager.

The fifth objective asks how organizations benefit from Google Cloud Customer Care. Google describes Customer Care as providing access to general how-to advice, best practice help, troubleshooting and operational knowledge through its support services, and there are four of them, matched to how much is at stake. Basic Support is included for all Google Cloud customers and provides documentation, community support, Cloud Billing support and Active Assist recommendations. Standard Support is recommended for small to medium organizations with workloads under development, looking to begin their support journey. Enhanced Support offers faster response times and additional services for medium to large companies running their cloud workloads in production. Premium Support is designed for enterprises that run priority workloads and require fast response times, platform stability and increased operational efficiencies — and Premium customers are assigned a Technical Account Manager, often shortened to TAM. The benefit to a cloud adoption is fit: an organization pays for the level of support its workloads actually need, and moves up as those workloads move into production. The exam tests the fit, never the price, and this deck quotes no prices.

The life of a support case

From opening to closing, and the one way back

Figure: A flow chart of a support case's statuses: opened with a priority from P1 to P4, then new, assigned to a specialist, in progress, sometimes waiting on the customer and back, then solution offered and closed. A dashed arrow shows that a closed case can be reopened within fifteen days.

Worked example (synthetic). A team replies to a closed case three weeks later. It cannot be reopened — the window is fifteen days — so the team opens a new case.

The sixth objective is the life of a support case. A support case is a request for technical assistance or help with a Google Cloud account, project or billing issue, and before creating one Google suggests reviewing known issues to see if a case has already been filed. When you open it, you choose a priority, and the Customer Care team responds based on that priority and on the support service that applies. Then the case moves through statuses. New means the case is not assigned yet. Assigned means it is with one of Google's specialists. In progress means specialists, or Google engineers, are working on it. Waiting on customer means Google needs more information from you before it can proceed — which is why Google's advice is that the best support cases are detailed and specific, saying what happened and what you expected to happen. Solution offered means a solution has been offered, and Closed means the case is resolved and closed by the specialist. There is one way back: you can only reopen a case that has been closed in the last fifteen days. After that, it is a new case.

Priority, and when to escalate

Raise the priority for impact; escalate for a broken process

Worked example (synthetic). A team escalates a P3 case because production has just gone down. The better move is raising it to P1 — escalation does not generally make a high-impact case move faster.

Priority is the lever a customer controls, so learn Google's four definitions. P1 means the application or infrastructure is unusable in production, with a significant rate of user-facing errors. P2 means the infrastructure is degraded in production, with a noticeable rate of errors or difficulty spinning up new production systems. P3 means the issue is limited in scope or severity. P4 means little to no business or technical impact. Because P1 cases are so serious, they follow the sun by default — handed off between global teams for twenty-four-hour coverage. If the impact changes, Google says you can change the priority based on urgency and business impact. Escalation is a different tool with a different purpose: it is meant to flag increased business impact or a breakdown of the resolution process, and an escalated case is immediately assigned to an Escalation Manager. The trap is using escalation to go faster. Google is explicit that when you are experiencing high-impact issues, escalation does not generally make the case proceed faster — the right move is the right priority.

What this topic actually tests

Four discriminations, six objectives

Metric, log or trace? a number over time, an event record, or one request's path. RTO or RPO? how long the switch takes, or how much data the lag can lose. SLI, SLO or SLA? the measurement, the target, or the contract — with the error budget in between. Priority or escalation? priority for impact; escalation for a broken process.

Close the topic on the four discriminations it tests. First, telemetry: metric data is a number measured at regular intervals, log data is a record of activity over time, and trace data is the path of one request — Cloud Monitoring, Cloud Logging and Cloud Trace respectively. Second, disaster recovery: the recovery time objective is how long the application may be offline, the recovery point objective is how much data may be lost, and they sit on different arrows. Third, reliability terms: the service-level indicator, or SLI, is the measurement, the service-level objective, or SLO, is the target below one hundred percent, the service-level agreement, or SLA, is the whole agreement with its penalties, and the error budget is the room that lets a team change things safely. Fourth, support: pick the priority for the impact, and escalate only when the process itself has broken down. The next topic, sustainability, closes the unit.

Official sources for this topic

• Cloud Digital Leader exam guide — Section 6
• Well-Architected Framework — Operational excellence
• Cloud Monitoring overview
• Google Cloud Observability documentation
• Cloud Logging overview
• Cloud Trace overview
• Well-Architected Framework — Reliability
• Well-Architected Framework — Reliability principles
• Architecting disaster recovery for cloud infrastructure outages
• Concepts in service monitoring — Cloud Monitoring
• Disaster recovery planning guide
• SRE fundamentals: SLIs, SLAs and SLOs — Google Cloud Blog
• Cloud Customer Care overview
• Customer Care procedures
• Best practices for working with Customer Care
• Well-Architected Framework — Manage incidents and problems
• DevOps capabilities — Google Cloud