A Postmortem Template for Nettle

Postmortem: YYYY-MM-DD - Incident

Postmortem Owner: John Doe |  Incident Runner: Jane Doe
Start Time: YYYY-MM-DD HH:MM:SS TZ | End Time: YYYY-MM-DD HH:MM:SS TZ | Duration:
Postmortem Review Date: Today’s date | Postmortem Verification Date: Future Date of Verification Meeting

Executive Summary

Cover in this section what happened, why it happened, how long was the outage, what fixed it and what the action items are. 


A post-mortem review is done in 2 meetings - one when the Postmortem is reviewed and discussed to figure out the right root cause (owner: Postmortem-lead or delegate) and two when the implementation of the AIs from the Postmortem are verified as implemented and deployed to production (owner: SRE-lead or delegate). The second meeting should happen no later than 1-2 weeks from the postmortem review date. The process of a postmortem needs to be tracked in a ticket. The first item of the post-mortem review meeting is to schedule the verification meeting and ticket.

Verification

Review Meeting Date: YYYY-MM-DD T HH:MM:SS

Review Meeting Scheduled? Yes / No - please schedule the meeting right now

Review Meeting Owner: 

Review Meeting Verification Ticket: 

Timeline

The timeline must begin with the first related change that impacted whatever happened in production. The timeline should end with the last remediation step at which 100% recovery was verified. In between, identify changes to cluster metrics and monitoring that were observed and how the team made changes. 

ISO Date/Time with TZ

What Happened





Incident Statistics

Time to Detect


Notes

Time to First Action



Time to Remediation



Time to Full Recovery




5-Whys

We run blameless postmortems. Please indicate factually why something happened with a reference to the actual defect or symptom. Reference commits, deploys and manual changes. It is ok and encouraged to mention people as long as it’s in the right spirit and in a neutral tone. We provide trust by default, psychological safety and generate helpfulness in this process. Our goal is continuous improvement. 

  1. App.nettle.llc went down. Why?

    1. Because build B was deployed

  2. Why?

    1. Because commit C included a bug in feature X.

  3. Why?

    1. Because issue C was not detected by person P as part of the build process. 

  4. Why?

    1. Because E was not functional

  5. Why?

    1. Because missing test E

Remediation Action Items and Owners

Focus on remediation items that will prevent the incident from happening again. 

Priority

Fix By

Owner

Action Item




Ticket Link with brief description





All remediation items will be expected fixed by the review meeting. Postmortem owner owns the responsibility of ensuring completion of the action items before the review meeting.

Comments

Post a Comment

Popular posts from this blog

Sadi Gali - Punjabi Lyrics and Meaning (in English) - Tanu Weds Manu

Image
Another superb song makes its appearance on the Bollywood scene. Sadi Gali from Tanu Weds Manu is a perfect Punjabi Dance track. Performed by Lehmber Husainpuri from RDB, words aren't sufficient to describe it. Just listening to it makes you want to jump up and start tapping your feet to the music. It's even garnered over a million plays on Youtube. In an effort to make this song's lyrics accessible to the masses, here's the meaning of the song in English paragraph by paragraph. I made it while listening to this song non-stop on loop. Hope you enjoy it!   Kudiyan de vich phir hassdi khed di [Hasdi] x 4 Khed di Ho, gutt di parandi teri naag wangu meldi [Naagawa-] x 4 -ngu meldi Again you're smiling and playing in the midst of girls Ho, Your "gutt da paranda"* resembles the moves of a snake gutt da paranda = An artificial hair extender often worn by Punjabi women to increase the length of their locks. Kudiyan de vich phire hassdi khed di College nu j...
Read more

Coke Studio: Madari English Meaning and Lyrics

Image
Thanks to Sanat Rath , I had the absolute privilege of listening to this superb rendering of Madari by Vishal Dadlani and Sonu Kakkar and an absolutely stunning composition by Clinton Cerejo as part of Coke Studio. Here's the official video. The lyrics of the song are interspersed with the meanings translated to English. Credits for the superb lyrics go to Manoj Yadav. Hope you enjoy it. Make sure you watch this in HD, it's worth the extra audio quality. Song: Madari Producers: Coke Studio Singers: Vishal Dadlani and Sonu Kakkar (Intro music with chords) (Slow build up) (Sharp guitar riff) (Drums kick in...) [Vishal Dadlani sings] Madari, Madari, Madari, Madari mera tu, Main Jamura re Jamoora,  Jamura re Jamoora...  Master, master, master * you're my master I'm an obedient assistant * , an obedient assistant... Madari, Madari, Madari, Madari mera tu... Main Jamura re Jamoora,  Jamura re Jamoora re...  Master, master, m...
Read more

AJAX और हिंदी

हाँ! अब आप हिंदी में भी भी ब्लोग कर सकते हैं। गूगल में तो कुछ अलग ही बात है। इतनी आसानी से मैंने कभी हिंदी नही लिक्खी थी। चलो, इसी बहाने मेरी हिंदी कुछ और शुद्ध हो जायेगी। इस लेख का मुख्य उद्देश्य है आपको AJAX नामक टेक्नोलॉजी (इसे हिंदी में क्या कहते हैं?) के एक अंग से परिचित करवाना। AJAX का पूर्ण नाम है Asynchronous Javascript and XML और इसी के मध्यम से मैं अभी इन्टरनेट के द्वारा ब्लॉगर के इस सुन्दर यन्त्र का फूलपूर्वक इस्तमाल कर रह हूँ। दरसल AJAX द्वारा हम Desktop जैसी look aur feel web applications को दे सकते हैं। पर आज के लिए बस इतना ही। अगले लेख में मैं इस टेक्नोलॉजी पर ज्यादा विस्तार से बात करूंगा। तब तक के लिए शुभ रात्री, शब्बा खैर, और अपना ख़याल जरूर रखियेगा। तब तक के लिए नमस्कार। ;-)
Read more

Tune Meri Jaana Kabhi Nahin Jaana - Lonely (Emptiness) - IIT Guwahati - Rohan Rathore

Image
Update: Rohan Rathore and his tragic story is a hoax for sure. This is confirmed. I've talked to my friends at IIT Guwahati and there is no record of such a person ever existing on campus. Nobody at IIT-G knew him and if a person dies in tragic circumstances, nearly everyone in the IIT knows it. The story about Rohan Rathore was created in order to gain views for a video on Youtube. So let us all put the "fake" Rohan Rathore to rest permanently as a person who never existed. Therefore, we can be very sure that the original singer of the song is Gajendra Verma. I'm leaving the original post below for archival purposes. ------ Original Post follows ---- I was introduced to this song by Sanat nearly a month ago and today, after listening to it again on Youtube,  I really felt that I needed to transcribe its lyrics. It's such a soulful song that you can't help feeling sentimental after hearing it - especially since the IITian (Rohan Rathod) who wrote a...
Read more

Voting Rights: Anomaly Free Boards

The following are the only anomaly free Board structures possible for board sizes 1, 4, 7, 11 and 13. Note that Corporate Governance research says that anomaly free boards are in 4D and 11D+ and anything in between doesn't work (because of the obstruction from exotic S7 which can embed only in R11 (flat) and R12 torsioned). We fix that by taking the 7 member board and mapping it to 11D. [See the discussion on Stable Homotopy Groups of Spheres in the appendix].  Please double check the case of 13 (only 12 people must vote on any issue, 1 must be the observer and the special resolution pass threshold is 9/12 and the budget pass threshold is less than 5/12 unified no-votes and 7/12 yes votes, abstains don't sink the budget vote. [There's a more nuanced variation using the Binary Golay Code voting at the bottom of this document; the ultimate bottom of this document is an attempt by Gemini to create something that works, I'm not sure how well it did, but it does maintain som...
Read more