Site Reliability Engineering - Part 1: SRE and Organizational Culture
Published at 2023-08-18T22:43:47+03:00
I work as a Site Reliability Engineer (SRE). In this blog series, I want to write down what SRE is about from my point of view. It's a mix of tech, culture and a lot of persistence, and there's never a boring day. This first part is about SRE and organizational culture.
2023-08-18 Site Reliability Engineering - Part 1: SRE and Organizational Culture (You are currently reading this)
2023-11-19 Site Reliability Engineering - Part 2: Operational Balance
2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture
2024-09-07 Site Reliability Engineering - Part 4: Onboarding for On-Call Engineers
2026-03-01 Site Reliability Engineering - Part 5: System Design, Incidents, and Learning
▓▓▓▓░░
DC on fire:
▓▓ ▓▓ ▓▓
░░ ░░ ▓▓▓▓ ██ ░░ ▓▓▓▓ ▓▓
▓▓░░░░ ░░ ▓▓▓▓ ▓▓░░ ▓▓▓▓
░░░░ ▓▓▓▓▓▓ ▓▓ ▓▓ ▓▓ ▓▓▓▓▓▓ ▓▓
▓▓░░ ▓▓▒▒▒▒▓▓▓▓ ▓▓ ▓▓▓▓ ▓▓▓▓▓▓ ▓▓▒▒▒▒▓▓▓▓ ▓▓▓▓
██▓▓ ▓▓▒▒░░▒▒▓▓ ▓▓██ ▓▓▓▓▓▓ ▓▓▒▒▓▓ ▓▓▒▒░░▒▒▓▓ ██▓▓▓▓
▓▓▓▓██ ▓▓▒▒░░░░▒▒▓▓ ▓▓▓▓ ▓▓▒▒▒▒▓▓ ▓▓▒▒░░▒▒▓▓██▓▓ ▓▓▒▒░░░░▒▒▓▓ ▓▓▒▒▒▒▓▓
▓▓▒▒▒▒▓▓▓▓▒▒░░▒▒▓▓▓▓▓▓▒▒▒▒▓▓ ▓▓▓▓░░▒▒▓▓ ▓▓▒▒░░▒▒▓▓▒▒▒▒▓▓ ▓▓▒▒░░▒▒▓▓▓▓▓▓▓▓░░▒▒▓▓
▒▒░░▒▒▓▓▓▓▒▒░░▒▒▓▓▓▓▒▒░░▒▒▓▓ ▓▓▒▒░░▒▒▓▓ ▓▓░░░░▒▒▒▒░░░░▒▒██████▒▒░░▒▒██▓▓▓▓▒▒░░▒▒▓▓██
░░░░▒▒▓▓▒▒░░▒▒▓▓▓▓▓▓▒▒░░▒▒▓▓██▒▒░░░░▒▒▓▓ ▓▓▒▒░░▒▒▓▓▒▒▒▒░░▒▒▓▓▓▓▒▒░░▒▒▓▓▓▓▓▓▒▒░░░░▒▒▓▓▓▓
░░░░▒▒▓▓▒▒░░░░▓▓██▒▒░░░░▒▒▓▓██▒▒░░░░▒▒██▓▓▓▓▒▒░░▒▒▓▓▓▓▒▒░░░░▒▒▓▓▒▒░░░░██▓▓▓▓▒▒░░░░▒▒████
▒▒░░▒▒▓▓▓▓░░░░▒▒▓▓▒▒▒▒░░░░▒▒▓▓▓▓▒▒░░░░▒▒▓▓▓▓▒▒░░░░▒▒▓▓▒▒░░▒▒▓▓▓▓▓▓░░░░▒▒▓▓▓▓▓▓▒▒░░░░▒▒▓▓
▒▒░░▒▒▓▓▒▒▒▒░░▒▒██▒▒▒▒░░▒▒▒▒██▒▒▒▒░░░░░░▒▒▓▓▒▒░░░░▒▒▒▒░░░░▒▒████▒▒▒▒░░▒▒██▓▓▒▒▒▒░░░░░░▒▒
░░░░░░▒▒░░░░░░░░▒▒▒▒▒▒░░░░▒▒▒▒▒▒░░░░░░░░▒▒▒▒░░░░░░▒▒▒▒░░░░░░▒▒▒▒░░░░░░░░▒▒▒▒▒▒░░░░░░░░▒▒
░░░░░░░░░░▒▒░░░░░░░░░░░░░░░░░░░░░░░░▒▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▒▒░░░░░░░░░░░░░░░░░░
SRE and organizational culture
At the core of SRE is prevention over cure. Traditional IT ops mostly react to problems. SRE tries to spot them before they happen, with Service Level Indicators (SLIs) and Service Level Objectives (SLOs): concrete metrics and targets for how reliable a service has to be for its users.
Another key concept in SRE is the "error budget." No system is perfect and failures will happen, so instead of pretending otherwise, teams get a budget for errors. Mistakes aren't punished; they're something to learn from.
SRE is also about people. One thing it tackles is the hero culture which burns out so many IT teams. A hero who swoops in and saves the day is great once, but relying on that isn't sustainable. SRE puts the team first: shared knowledge, a balanced on-call rotation, trust and ownership. I've been there myself, falling into the hero trap, and I know firsthand that it's just not feasible to be the go-to person for every problem that comes up.
SRE also puts a big emphasis on documentation. Docs should be reviewed like code. Good docs make onboarding new team members much easier.
Adopting SRE can be a big challenge for some organizations. They might think the SRE approach goes against their goals, like preferring to roll out new features quickly rather than focusing on reliability, or seeing SRE practices as too much hassle. Building an SRE culture often means taking the time to explain things patiently and showing the benefits, like faster release cycles and a better user experience.
Monitoring and observability are big parts of SRE, so you need good tools to query and analyze the data. And SREs need to be curious; there's always some strange issue to dig into and some new tool to learn.
For SRE to work, everyone in the organization needs to buy in. Reliability is a shared responsibility. It doesn't work when the SREs are just the people who patch things up after the fact.
Organizations that have SLIs, SLOs, and error budgets in place are already pretty far along in their SRE journey. Getting there takes a lot of communication, convincing people, and patience.
Continue with the second part of this series:
2023-11-19 Site Reliability Engineering - Part 2: Operational Balance
E-Mail your comments to paul@nospam.buetow.org :-)
Back to the main site