Show logo
Explore all episodes

How Should We Handle Failure?

  |  Compiler Team  
Professional development

Compiler • • How Should We Handle Failure? | Compiler

How Should We Handle Failure? | Compiler

About the episode

In tech, there's a lot of pressure to keep things running smoothly. That makes even a small mistake or a brief outage seem disastrous. When people fail at something, they can experience many different emotions: Anger, regret, or even fear. Compiler speaks with guests from enterprise IT and beyond about how we should handle failure, and how teams and individuals can benefit from processing it safely and effectively.

Compiler team Red Hat original show

Subscribe

Subscribe here:

Listen on Apple Podcasts Listen on Spotify Subscribe via RSS Feed

Transcript

Hey, this is Kim. I'm one of the producers for Compiler. Before I bring in Brent and Angela, I wanted to talk to you about something a little personal. It's about failure; specifically, my biggest failure. I've worked in digital media for a long time, so I decided to take a course in user experience (or UX). It was towards a certification, and one of the classes was Introduction to Python. Now, this class gave me quite a bit of trouble. And when I say that, I'm being pretty generous because I didn't understand a single thing that was said to me. I couldn't get the simplest commands to work. I couldn't get anything to work, and ultimately, I failed the class. And I don't fail things, not often. Fast forward to now, I've been at Red Hat for, well, more than two years now. It's full of smart people, people who do code every day, and I couldn't even get a Hello World command to work. And while I'm good at my job, I always come back to that Python class. How do I move forward from that shortcoming? How should I handle that type of failure? This is Compiler, an original podcast from Red Hat. We're your hosts. I'm Angela Andrews. And I'm Brent Simoneaux. We're here to break down questions from the tech industry: big, small, and sometimes strange. Each episode, we get perspectives on where things are and where they're headed. Today's question: How should we handle failure? Producer Kim Huang is here to get things started. Angela, Brent, I want to talk today about failure. We're going to hear from a few people who are going to help us gauge what failure really means for people, personally and also what it means for teams when they work together. Before we get into that though, I want to know, what do you think about failure? Oh, well, failure happens to all of us. It's a part of life, and I've had my own personal failures. When I think back to college, I failed calculus, and that means I had to change my major. That was a personal failure for myself. I really don't know where my career would have gone had I stuck with my computer science major, but I wonder. I wonder if failure stings more when the thing you're failing at, is part of your identity or part of the person that you imagined you want to be. Yes. I think that that's very accurate. I read a lot of research going into the story, and what you said, Brent, reminded me of some studies that I read from this doctor from Northwestern University. His name is Dashun Wang, and he does research into success and failure. He talks about success being like a collective phenomena. And that means, in order for you to be successful, everyone has to agree, not just you. Everyone else around you has to agree that you were successful in whatever you were doing. Oh. I never thought of it that way. I mean, collectively, what people think of you is what you are, to some degree? Yeah. If everyone thinks you're successful, then you are successful? And if everyone thinks you're a failure, they think you are a failure? That's a weird dynamic to even consider, putting that much weight. Look at me saying this out loud, putting that much weight on what other people think of you as a way to define yourself. But that's interesting. What else do you have for us? I'm very curious. You have my full attention. Yeah. Let's dig into this, Kim. So who did you talk to first? Well, I wanted to know how people look at failure, so I spoke with someone we know pretty well. She was a Red Hatter, and she did a lot of work actually in early seasons of Command Line Heroes, dealing specifically with failure. She told me the way that people deal with failure personally translates into how they process failure on teams and how we all treat each other in the process. I'm Jen Krieger, and currently, I'm working at GitHub as the senior director of the Technical Program Management team, and I am just super excited to be here. She had a lot to say about how failure was framed in her working life. Failure was the focus. Failure was the thing that you led from in every situation. In fact, if I even look earlier in my career, when I think about that word, every single framework, technique, all that stuff that I learned, was mainly in place to prevent teams from failing. And so there was never really a conversation so much about why we were so focused on the negative part and less on the outcome of failure, which was to learn from our mistakes. And it seems like the industry still hasn't quite gotten there yet. We still focus on this concept of failure and safety and making it safe to fail. When I actually prefer to think about it a lot differently, which is making it safe to learn, which is a way different and more positive way of looking at that word. I have a lot of things going through my head right now. Okay. Making it safe to fail is something that I hear a lot. And I have actually never heard this phrase, making it safe to learn. Yeah. I've never heard that phrase used before. It's always, it's safe to fail, or fail fast. Yeah. Fail fast, yes. That's the phrase we always hear, but the safety, placing it on the positive, that's a different twist on it. Jen sees a lot of development teams go through really hard challenges. Many of them, very inherent in the constant cycles of production. For these teams, failure denotes chaos and fear. I keep looking at this when I think about software development. And I think about the fact that early in my career, folks were so incredibly afraid of a production outage. That was the thing that you would get paged about, that would basically destroy your weekend, your life for a period of time. So, you have all these forces. You have failure, you have fear, you have personal narratives, you have team failure and they’re all kind of like, to me, they're kind of like swirling around in a whirlpool of negativity. Is it just me? No, it's not just you. What you said had me thinking. I mean, it happens sometimes. Sometimes there's an outage, and we hear about it, talk about folks getting fired and heads will roll. And I think that's knee-jerk because someone needs someone to blame. Right? And when we think about what Jen said, we're making the team feel safe to learn. I'm sure in that postmortem, people looked at what happened, and they said, they figured out what the failure was, and they worked hard to remediate it, to make sure that it never happens again. This is why she's focused on creating a safe environment for those teams to fail and learn at the same time. When I say, make it safe to learn, that's really what I mean. It's not just about, "We have deployed code to production and production is down so, therefore, we are bad because we have failed." It's more, "How can we actually deploy code to production in a way that will help us be more resilient to the failures we've experienced in the past?" That's what I mean, making it safe to learn. Because the world is a very, very different place than it was even a few years ago, Jen says, safety– particularly psychological safety–will play an important role in failure as newer technology develops. People cannot learn these new technologies fast enough. They cannot adapt to the things that they're seeing grow out of these new technologies. At the end of the day, we don't always know the answers to the questions we're being asked to solve right now. And so, if we are constantly focused on this idea that failure is the thing that we need to make better for folks, we need to make it safe to fail. We need to make sure that people understand that they bring production down, they're not going to get fired. That's probably a good idea to do, but at the same time, companies that are just simply setting that bar there, that's not enough. They have to go even further than that. They have to make space for people to learn from those mistakes that they've made, and I just don't see the industry actually doing that. Oh, that is really interesting. And that's something that's... I don't know what the word is, maybe it's bothers or intrigued me about this phrase "failing fast" or making it safe to fail, the focus on failure. It's not the failing that we're striving toward. Right? Exactly. Mm-hmm (affirmative). It's the learning that comes from the failing that we're striving toward. And I think that's what Jen is saying, right? Yes. You two may recall there is a very famous story about Thomas Edison. And Jen, I think at some point, does mention in our discussion about Thomas Edison going through different iterations of the light bulb, and someone pointed out to him that he had failed a thousand times, but instead of saying that, he reframed it and said, "No, I just found a thousand ways that didn't work, and one that did." Mm-hmm (affirmative). That sounds very familiar. And when we hear that word failure, when we reframe it, when those things happen, because bad things are always going to happen, like she said, you need that psychological safety because that's where the growth happens, that's where the learning really happens in those moments. Because if nothing happens, where is the growth? What are you learning from exactly? Right? Right. I think the other thing is that failure sometimes does have real world effects. You know? Mm-hmm (affirmative). It actually does… like outages and things like that. Yes. So I'm trying to tease out the scale of failure and the effect of failure, and how you create safety and learning at all those different scales. That's interesting because, ultimately, companies want to provide their services and their products to customers in a timely manner because they're all competing with each other for attention, for a market share. So, I think this is why Jen is very skeptical as to whether or not companies will make failure safer. Let's hear from her. There's oftentimes not the organizational muscle to do anything about what they are meant to learn, so it's not just learning but also acting upon those learnings. I don't see organizations giving people the emotional space to actually be able to digest what just happened. People are not given permission to feel disappointment. For Jen, I think it's a really simple equation: a team can totally fail one day and then bounce back and put out an amazing product or do something really amazing. Yeah, we have technical failures. But the thing that I have seen from technical failures is, the technical teams that I've worked with over these years have always recovered. They've always found a way to fix the problem that they're having. But one thing that has not come back easily: trust. Mm-hmm (affirmative). What has not always happened out of these events has been, the relationship damage has been fixed, the teams have figured out how to work together moving forward. Maybe a finger pointing event that might have been visible to more than what would be an acceptable limit of people, no one ever apologizes for those things, and so a lot of that is embedded in your ego and the way that you perceive others. And also what I see mainly today, is the reluctance to actually get on the phone with somebody and talk through something. We are so reliant on texting and text-based form of communication. People are really reluctant to have a hard conversation with each other, and that's got to change. Sometimes, in workrooms, there is time for apologies. There's no time for hard conversations when you have deployments going out one after the other. But ultimately, how people process failure as individuals does matter to a team, especially in a fast-paced environment. If there's no time to be retrospective, how are people supposed to process what happened? In tech, we focus a lot on team dynamics and team outlooks and visions and vision statements and goals. But ultimately, teams are made up of people, and how people process failure as individuals does matter to a team, especially in a fast-paced environment when you’ve got one thing going out after the other, so many different deployments, and there's no time to react or be retrospective. This is part of the culture of the team, and I think it's part of how the team interacts with each other. I've actually worked with Jen in the past, and I've actually done some of her exercises with my team itself where she sort of holds space for everyone to digest what has happened, all the good stuff, all the bad stuff, the stuff that didn't go so well. And I love what you're saying there, Kim, because I don't think we hold enough of that space for ourselves. Okay. So, we just talked a lot about perfection and failure, and I was curious about whether or not failure had anything to do with a lack of knowledge or skill or passion. Yeah. But what I ended up learning from this next person that we have on the show, was a little bit more insight into how important it is to fail. Hello. My name is Erika Hamden. I am a professor of astrophysics at the University of Arizona. And I spend most of my time building telescopes, which I have decided is the best job that somebody can possibly have. This is a new one for us. I don't think we had an astrophysicist on the show. Really? No, we have not. And I wanted to ask if either of you had ever heard of Dr. Hamden. No, I have not. I have not, no. Okay. A few years ago, Dr. Hamden gave this amazing TED talk called What it Takes to Launch a Telescope. Huh. The video has been watched online millions of times. If you speak with her, you'll learn that she's no stranger to failure. One of the projects I work on is called FIREBall. That's what I gave my TED talk about. And it's a balloon telescope, which is kind of a weird, interesting thing. It's a telescope that goes on a high altitude balloon, not on the ground or in space. And I've been working on it since 2008, so it's getting to be almost 15 years now. Get ready for this ride. So, FIREBall has turned out to be a project plagued with many ups and downs (mostly downs). It has been marked with multiple failures, technical snafus, mechanical failures. There was even a bird that flew into the balloon one time. The balloon had a hole in it one time, and it crashed. There were so many different failures. And in her TED talk, Dr. Hamden talks at length about these repeated failures. And she says, success as a sure thing is one of the most crucial things to reject when you're working on a project. Mm-hmm (affirmative). You have to be prepared for that eventuality of like, your experiment worked correctly, you did everything right, but you were wrong in your initial question, because that's always a real distinct possibility. Earlier in the episode, we talked about failure and success being considered collective phenomena. That means, if something is to be successful, then other people have to observe and agree. When you're talking about something as complex as the scientific process, that can come off as a little bit oversimplified. Dr. Hamden says, the successes of science can sometimes hide the many, many, many setbacks that people face. I mean, that process is a necessary part of getting to the thing that does work. So, I feel like if you're only reporting on the very final check, when all of the hard work has already been done, then you're really missing a key part of the story which is that, this is just a continuous process of messing up. That's a little bit of a public communications challenge because you don't want people to think that the mess ups mean that you're not reliable, the mess ups are actually a good thing. And reporting on them is a good thing too. That's like the transparency that–in an ideal process, that's how science should operate, where you can see what worked and what didn't for everybody's projects. We're always focused on that end, that win at the end. Mm-hmm (affirmative). This is funny, but it reminds me a lot of the Michael Jordan, Kobe Bryant quotes. In my gym, we have all these Kobe quotes in there. Aw. And this is something that I see on a constant basis. So, they attribute "failure" to their success because without it, they wouldn't be Jordan, they wouldn't be Kobe. Again, people don't look at that, but that's where all that good stuff happens. That's how you become great. That's how the science gets notoriety. Anything. Anything good comes, there had to be some setbacks and failures in there to get you there. That's what I'm hearing. Dr. Hamden juxtaposes failure with effort, which may be a good way to change the negative inner voice that pipes up when things don't go as planned. Everybody deals with failures of all sorts of kinds. And the idea that you should be ashamed of it or that you should keep it a secret, I think, is really discouraging. Because in my view, trying and failing, the important part of that is that you tried something, and that should be celebrated. You tried it, that's scary and good. And instead, people feel more shame about the failure part. And I think that just shuts off a conversation that's really important to have. With a very unhealthy approach to failure, a person could take that and turn it into a personal narrative. But what she has done, she's framing failure in a more collaborative way. She is talking about having a positive mindset. She's not thinking about success as a binary. It's interesting because we tell stories about ourselves, and teams tell stories about themselves. Yeah. But sometimes those narratives and those stories can take hold in your mind. Right? Yes. Which is I think what you're talking about, like framing, what just happened? How are you talking about that? And is that being lodged in your mind? Is that being lodged in the company's mind? Is that being lodged in your boss's mind? What is that story? Exactly. So Dr. Hamden has thoughts on teams too, and how someone could frame failure in a much healthier, more collaborative way. I think it's partly just in a mindset and how you kind of talk about it with the people around you. I think that, especially in dealing with teams and working on these big projects, it's really important that blame not necessarily be part of the kind of analysis of what went wrong, that blaming people for things doesn't make them do better. It just makes them feel bad about it. Ooh. Yes. Wow. Are you trying to help people learn or are you trying to make them feel bad? Yes. I've been in postmortems where you kind of feel that finger going around the room, and you just don't want it to land on you. That is so much stress. I'm imagining everyone around that table, just sort of slumping down in their seats, trying to avoid eye contact. Yeah. That is so true. And when you're in that kind of hot seat, you start building that personal narrative again. Am I going to suffer any negative consequences from this? Am I going to be trusted if I'm a lead? Am I going to be trusted with leading another project? Yeah. Am I going to be trusted with my job? Am I going to lose my job? It's like an avalanche, kind of. Mm-hmm (affirmative). So, there is a binary we've established, success and failure. That's typically what this looks like. Yeah. Dr. Hamden has a different way of thinking about that binary. I think that the opposite of failure is inaction. So I view failure as a positive thing, that you tried something. And in my experience, most of the time when you try something, it doesn't work the first time, and that's fine. So, to me, I associate failure very strongly with just trying or the act of giving something a shot. So the opposite of that to me is not doing anything, it's to not take the leap. So I asked Dr. Hamden about how our jobs–in STEM, outside of STEM–how are societies define success, and how reality is much different? I think it's usually defined as anything other than complete perfection in whatever the goal is. It's like this all encompassing thing that there's one way to get success and there's a million ways to fail, and all of the million ways are bad. And I think that that is problematic and that the million ways to fail, there's probably a few of them that are bad. You failed because you didn't try. You failed because you sabotage this thing. Those are not ideal, but there's a lot of things that like, you failed because you're working with incomplete information. You failed because you're trying something new for the first time that you don't really understand, and now you understand it a little better. You failed because you are the first person in the entire history of the world who is doing something. All of those failures are really helpful. They're super valuable. I actually think that a string of failures is necessary for success. So, this sounds a lot like the Thomas Edison analogy story that you told us earlier where that's where the discovery happened. He found a thousand ways to do it the wrong way to get to the right way. That's an amazing way of thinking about failure. It really turns it on its head, and it takes that sting away from it. Listening to this episode, it was actually a little emotional where I'm trying to internalize the amazing things that people are saying here. Yeah. Because sometimes, people can be very hard on themselves. So, can failure lead to discovery? One would hope, because as long as you don't give up and you keep trying and you do better and you iterate, the process gets better. You get better, you learn more. So when you say it like that, that's where the good stuff happens. Mm-hmm (affirmative). I think we're hard on ourselves. Ooh. And I think we're hard on other people sometimes too. Absolutely. And I think one of the big things that I'm taking away from this, is the spaces that we create for ourselves, but also for our teams and also for people that we work with every single day, how do we create that space where learning can occur? The way that we process failure needs space, first and foremost. I think that's what Jen was getting at. You need that space to process what happened. I think that has an impact on how teams work together because teams are just people, gathered together for a shared purpose or vision. That can also affect how individuals themselves learn from failure. If they feel like they're not good enough, if they don't have time and space to process failure, that all could come together in a way where they look at their work in a negative light. And while looking back is good, you can't keep looking back. We all have to move forward from our missteps. We have to not make our professional failures into stories about who we are as people. In our working lives, we could use failure to discover things that we didn't know before, learn new things, go new places with our talent, push ourselves beyond what we thought was possible. We can also lift ourselves up and the other people around us as well whenever we run into a difficult situation. I think that's the most beautiful part about what I learned. Failing that Python class doesn't say anything about me personally, and it doesn't help me at all to create that story about myself. It just means I was doing something new. And when you're trying something new, and you fail, you can always try again. And that does it for this episode of Compiler. Today's episode was produced by Kim Huang and Caroline Creaghead. Victoria Lawton always sticks with us no matter how many times we mess up. Our audio engineer is Elisabeth Hart. Special thanks to Shawn Cole. Our theme song was composed by Mary Ancheta. Big thank you to Dr. Erika Hamden and Jen Krieger for sharing their perspectives with us. Our audio team includes Leigh Day, Laura Barnes, Claire Allison, Nick Burns, Aaron Williamson, Karen King, Boo Boo Howse, Rachel Ertel, Mike Compton, Ocean Matthews, and Laura Walters. If you liked the show, we would love for you to come back next time, so subscribe, leave us a review. It really does help us out. We'll see you soon. Thanks for listening. All right. See you next time.

More about professional development

Keeping up with changes in technology requires training and and engagement with learning opportunities—either in a traditional setting, or individually.

Blog Post

Red Hat Enterprise Linux Common Criteria and FIPS certificates

August 18, 2025  |  Jaroslav Reznik

...

Blog Post

Friday Five — August 15, 2025 | Red Hat

August 15, 2025  |  Red Hat Corporate Communications

...

Technically Speaking, an original podcast from Red Hat

Original Shows

Technically Speaking | The role of the OS in the Age of AI

Artificial intelligence
Professional development
Compiler, an original podcast from Red Hat

Original Shows

Conferences 102 | Compiler

Open source
Professional development
Compiler, an original podcast from Red Hat

Original Shows

Rolling with the Punches | Compiler: Tales From The Database

Application development and delivery
Culture and process
Professional development

About the show

Compiler

Do you want to stay on top of tech, but find you’re short on time? Compiler presents perspectives, topics, and insights from the industry—free from jargon and judgment. We want to discover where technology is headed beyond the headlines, and create a place for new IT professionals to learn, grow, and thrive. If you are enjoying the show, let us know, and use #CompilerPodcast to share our episodes.