Site reliability engineering

The field of site reliability engineering originated at Google with Ben Treynor Sloss,[5][6] who founded a site reliability team after joining the company in 2003.[7] In 2016, Google employed more than 1,000 site reliability engineers.[8] After originating at Google in 2003, the concept spread into the broader software development industry, and other companies subsequently began to employ site reliability engineers.[9] The position is more common at larger web companies, as small companies often do not operate at a scale that would require dedicated SREs.[9] Organizations that have adopted the concept include Airbnb, Dropbox, IBM,[10] LinkedIn, Netflix,[8] and Wikimedia.[11] According to a 2021 report by the DevOps Institute, 22% of organizations in a survey of 2,000 respondents had adopted the SRE model.[12][13]

Site reliability engineering, as a job role, may be performed by individual contributors or organized in teams, responsible for a combination of the following within a broader engineering organization: System availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning.[14] Site reliability engineers often have backgrounds in software engineering, system engineering, or system administration.[15] Focuses of SRE include automation, system design, and improvements to system resilience.[15]

Site reliability engineering, as a set of principles and practices, can be performed by anyone. SRE is similar to security engineering in that everyone is expected to contribute to good security practices, but a company may decide to eventually hire staff specialists for the job. Conversely, for securing internet systems, companies may hire security engineers. To define and ensure their reliability goals, companies may hire SREs as well.

Site reliability engineering has also been described as a specific implementation of DevOps, although they are slightly different. SRE focuses specifically on building reliable systems, whereas DevOps is more broadly focused.[2][3][4] Although they have different focuses, some companies have rebranded their operations teams to SRE teams with little meaningful change.[9]

There have been multiple attempts to define a canonical list of site reliability engineering principles, but while consensus is lacking, the following characteristics are usually included in most definitions:[1][16]

• Automation or elimination of anything repetitive in a cost-effective way.
• Avoidance to pursue much more reliability than what's strictly necessary. Defining what's necessary is a practice by itself (see list of practices below).
• Systems designed with a bias toward the reduction of risks to availability, latency, and efficiency.
• Observability—as in, the ability to ask arbitrary questions about a system without having to know ahead of time what to ask.[17]

The site reliability engineering practices also vary widely, but the list below is relatively commonly seen as at least partially implemented:

• Toil management as the implementation of the first principle outlined above.
• Defining and measuring reliability goals—SLIs, SLOs, and error budgets.
• Non-Abstract Large Scale Systems Design (NALSD) with a focus on reliability.
• Designing for and implementing observability.
• Defining, testing, and running an incident management process.
• Capacity planning.
• Change and release management, including CI/CD.
• Chaos engineering.

Site reliability engineering teams engage with the other teams within their companies and the SRE principles and practices in various forms. Here is a high-level overview of common SRE team implementations:[18]

The USENIX organization has held an annual SREcon conference since 2014 for site reliability engineers in the industry, and also holds regional conferences with similar themes.[19]

• Limoncelli, Tom; Chalup, Strata R.; Hogan, Christina J. (September 2014). The Practice of Cloud System Administration: DevOps and SRE Practices for Web Services. Vol. 2. Upper Saddle River, NJ: Addison-Wesley. ISBN 978-0133478549. OCLC 891786231.
• Beyer, Betsy; Jones, Chris; Petoff, Jennifer; Murphy, Niall Richard, eds. (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly. ISBN 978-1491929124.
• Blank-Edelman, David N., ed. (2018). Seeking SRE: Conversations About Running Production Systems at Scale (1 ed.). Sebastopol, CA: O'Reilly. ISBN 978-1491978863. OCLC 1052565720.
• Beyer, Betsy; Murphy, Niall; Kawahara, Kent; Rensin, David; Thorne, Stephen (2018). The Site Reliability Workbook: Practical Ways to Implement SRE. O'Reilly. ISBN 978-1492029502.
• Welch, Nat (2018). Real-World SRE: The Survival Guide for Responding to a System Outage and Maximizing Uptime. Packt. ISBN 978-1788628884.
• Adkins, Heather; Beyer, Betsy; Blankinship, Paul; Lewandowski, Piotr; Oprea, Ana; Stubblefield, Adam (2020). Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems. O'Reilly. ISBN 978-1-4920-8312-2. OCLC 1129470292.
• Rosenthal, Jones, Casey, Nora (2020). Chaos Engineering: System Resiliency in Practice. O'Reilly. ISBN 978-1492043867.{{cite book}}: CS1 maint: multiple names: authors list (link)

• Awesome Site Reliability Engineering resources list
• How they SRE resources list
• SRE Weekly weekly newsletter devoted to SRE
• SRE at Google landing page for learning more about SRE in Google
• Komodor K8s Reliability learning center with resources for SREs working with Kubernetes