How Do We Make Updates Less Annoying?

  |  Compiler Team  
Automazione e gestione

Compiler • • How Do We Make Updates Less Annoying? | Compiler

How Do We Make Updates Less Annoying? | Compiler

About the episode

Updates. They take time out of your day and your devices out of commission. That’s about the extent of it for an end user. But for organizations? Updating their systems is a big deal. And forgoing regular updates is a recipe for disaster.

"If it ain’t broke, don’t fix it" isn’t a good option for enterprise IT. Your system may keep running. but if you haven’t updated in awhile, you may be missing some stability and performance improvements—and probably some security patches too. Regularly updating is a good habit. How do we make it less annoying?

Compiler team Red Hat original show

Iscriviti

Subscribe here:

Listen on Apple Podcasts Listen on Spotify Subscribe via RSS Feed

Trascrizione

Angela, Brent, how do you feel about updating your software? Hmm, well, it depends. Usually it's like, I'll do it as soon as I'm done for the day so I'm not really impacted, and I try to stay on top of it because I know it's important. I do my best. I think I'm okay with it. What about you, Brent? I am what I would describe as a smug updater. A smug updater? Oh, do tell me what that means. Tell me more. I think I understand the importance of updating and so I just feel really good about doing it. But not only that, for me, it feels like I'm being productive, but I don't really have to do much of anything. So I feel like I'm getting this like really important task done, but how much effort does it really take? I am a... I guess I'll call it a reluctant updater where, again, I know that it's super important to keep things up to date, but I get... I just get so annoyed by those notification messages saying like, "you've got a new update" and it's like, "ah, geez, I just did this. Do I have to do this again?" And then sometimes you roll out an update and it causes more problems than it fixes. All heck breaks loose. I know about those updates too. And this is all great for us as end users. But for businesses, updates are a much bigger deal. There are security implications, not just for your business, but for all of your customers as well. Right? Huge implications. That's right. And doing it right takes planning, doing it wrong has consequences, and not doing it at all is just a recipe for disaster. So. If updating is something that we have to do all the time, how do we make updates less annoying? All right. Let's dig in. This is Compiler, an original podcast from Red Hat. We're your hosts. I'm Brent Simoneaux. And I'm Angela Andrews. We're here to break down questions from the tech industry, big, small, and sometimes strange. Each episode, we go out in search of answers from Red Hatters and people they're connected to. Today's question: How do we make updates less annoying? Producer Johan Philippine is here with the latest update. So we're going to be talking to two people today, Ben Pritchett and Vania Vergara, they're both going to help us out a little bit with the update landscape. First up, Ben Pritchett. He just started at IBM, but up until very recently, he worked on upgrading some of Red Hat's internal infrastructure. And he helped me out at the top with some definitions. Now, Brent? Yes. We're going to do your favorite thing. We're going to get out the whiteboard real quick. I feel like every time we talk to you, Johan, we have a whiteboard in front of us. I always have a whiteboard in front of me. I know... I know you do. I like whiteboards. What can I say? They're useful. All right, let's do this. We're going to write X-dot-Y-dot-Z on the whiteboard. Okay. Now these letters together, they represent a software's version number, and how it's broken- X-dot-Y-dot-Z. That's right. Now, "X" all the way on the left. That's the big version number, major version. This one doesn't change very often. Think RHEL [Red Hat Enterprise Linux] 8 or RHEL 9, Windows 10 or Windows 11. These are big years-long updates that change a lot of the features and kind of completely change the architecture of, say, an operating system or the software and how it works. And often enough it's a big enough change that you kind of want a clean break from one installation to another. Angela, you can correct me if I'm wrong. There are ways to update from one to the other, but it can get pretty messy. Is that right? It can. Yes. The clean break option is always recommended. Would you say that it's an extreme update? I would, and so would you, and so would Ben, actually. I would generally summarize these as major re-architectures in the offering. Angela was right. Ben agrees. Now "Y" stream updates are that middle number. They're smaller updates, less of a big deal than the major version updates, and they're more frequent. Generally newer features. You could even see some deprecation of features as you kind of go through different "Y" stream versions. That's still going to be a pretty big change that you have to really plan for, right? Because if, if you want to use those new features, you’ve got to plan for integrating them to the rest of your software, the rest of your system. And if something is being deprecated that you're actively using, you have to figure out, "Well, okay, how am I going to replace this feature? How am I going to disconnect everything that previously was relying on it?" It can be a pretty big deal. So as a user, you should expect to see some difference here. Exactly. And you should also read the release notes for every update that comes out because you don't want to be caught by surprise. And as a user, you want to know what's coming on. But as a business, as an enterprise where your own infrastructure is running your application, that's running your website, you want to be absolutely sure that you know what's being added or released before you hit that update. Because if some, if something changes and you weren't aware of it, then your system could go down, then that's not good. All right, we've got our "X" stream. We've got our "Y" stream. What about the "Z" stream? The "Z" stream. So those are at the very end. They tend to be the most frequent updates. These are small updates that typically don't change the functionality of a system, but they're very important for other reasons. These are kind of the general, like bug fix or smaller feature updates. Much smaller in terms of the scale, but much more frequent and still very important. Right. So, for some of our listeners, maybe this X, Y, and Z stream may sound a little bit new to you, but if you're used to hearing major, minor and patch, that's basically the same thing we're talking about here. Just a different nomenclature. So we have those in mind, X stream, Y stream, Z stream, but rolling them out is still the issue. Vania Vergara, she's a technical account manager here at Red Hat. She gave me, let's go with the dream scenario for updating your infrastructure. Oh, the dream is to click a button. That's the dream. You push a button and then the update just kind of rolls out, and you don't have to think about it. And it's all easy. It just works easy-peasy. But that's not how this happens in real life though. No. It's an issue that is most common because we are always updating our customer's versions. So, we have a lot of problems there because sometimes the systems are really complex and to upgrade is not an easy task. So, it's easy for us as consumers to just push a button and then the iPhone update rolls out or your computer update rolls out and you don't worry about it. But on the other side, getting that update ready, writing it, rolling it out. That can be really complex because it's, again, in a lot of situations, it's not as easy as just pushing a button. Where does that complexity derive from? Well, because usually you are the person who's responsible for approving the updates to be rolled out to the infrastructure. You have to make sure that they do no harm. There's a lot of interoperability between systems and functions. So you have to test them and make sure that you don't break anything. Enterprise systems have a lot of components that really need to work together. So, yes, there is a lot that goes into it, research, understanding... It is a really complex operation and once you're okay, then you go ahead and approve it and you push that button. And now it gets dispersed through your environment. So Vania explained to me just how necessary regular updates are. Well, it's really important to update because, we have a lot of issues with security nowadays. And not only security issues, but if you want to have a system that is stable and secure, if you want to have a consistent infrastructure, you should have this updated. We talk a lot about security, but that's only really one aspect of keeping things up to date. That's true. And she talked about stability. So, they're not necessarily one and the same. Say if there is a security vulnerability, that's out there, it's out in the wild. You want to make sure that you've protected the systems in your environment from that vulnerability. Stability could be, like we said, like a bug fix where something really wonky is going on in a particular OS [operating system] or software. You want to make sure that the end users have a stable product that they're using. So, someone's gone through the work to try to fix something to make sure things work as they should. If you're that end user and you're using said product, and you're having all of these symptoms, you really do want your system to be stable. So, making sure your systems are both stable and secure is really important, if you're managing an infrastructure. I kind of have this feeling that maybe... a lot of organizations, maybe it is stable for them, or it feels stable, or it feels like everything's still working. And so the cost of updating might be worse than the perceived stability of the current system. Let's dive into that a little bit more all right. I asked Vania exactly about that same thing, about the "if it ain't broke, don't fix it" and how bad can that get? Famous last words. It turns out it can get pretty bad. They have some companies that have system 20 years old, 22 years old, they just don't touch it because it works and they don't have the people to interact with the system. They don't have the knowledge. So, if it works, leave it be. But you have a lot of issues of vulnerability with this system. And you have to take care of this. Boy, I felt this in my spirit because this is more common than you think. Yeah. Oh, yeah. Yeah. This is really an issue that a lot of companies and organizations have to deal with. They're running these ancient operating systems and bespoke software that someone wrote and have since moved on in one form or another, and it works. It's running something that's critical to the organization, but there's no knowledge on, on how to fix it if it does break. So what do you do? You leave it alone and cross your fingers and your toes. And you're saying this is not uncommon, right? Not in my experience. Yeah. Not in Vania's experience either. The more you wait... Difficult, it will be to upgrade. And sometimes it will be just, you can't do it. It will be just unadvisable to upgrade. You have to build a new system entirely. It will take more time, more effort, and more money. It takes engineering time. It takes money to buy the software licenses to buy the hardware upgrade. It can get very expensive to update these systems and even more so when you have to rebuild it from scratch. So, they have to make that decision and it's not an easy one to make. And sometimes that can get kicked down the road to say another manager, or there's been some change in an organization and you've come in and all you hear are the stories, right? Don't touch this system. There's no support for this system. No, it hasn't been updating, but it's still running X application. So, this isn't... I'm sure companies don't do this lightly, but it's a lot holding them back from making that decision. There's also all of the other work that your team is responsible for as well. All of that stuff doesn't go away. Ben was telling me that even when you know that he and his team were working on a major update, they still had their other job to do. There's never one thing going on in an IT Org. If I had to estimate it was roughly 50% of the engineering team's time focused on build-out of this new platform. Obviously we had to continue to support apps. So he had to split his time evenly between supporting existing infrastructure and building out the major update. Vania reiterates, why that 50% can take a while. So we don't have people to go through with the process of upgrading. Because some people believe or some specialists or professionals believe that an upgrade is just to click a button. So, I will upgrade my Windows. I will click a button and everything's okay. It's not quite the same. When you have complex systems or systems that are integrated with several platforms, different system- operational systems. If they have this kind of complexity, you have to test all the integrations. And if you operate one part of the system, maybe the other parts won't communicate themselves. And it's really annoying, and you don't want to discover it on production. She said annoying. Okay. Annoying. That's a nice way to put it. She was being a little diplomatic there, but yeah... She's not wrong. She is not wrong. I have another word for that, but annoying will do. It's annoying if you discover it before it rolls out into production, but if you discover it after it rolls out into production, then annoying is not the word that I would use either. But there's all these integrations that you have to test. Angela, you were telling us about this earlier. You have to make sure that all the different parts of your system still work and that they're communicating with each other. And if you don't get all that testing right, you could be in a lot of trouble. So many things that could go wrong. If you look from the application side, maybe you upgrade your system in the database wasn't updated. If you are updating a software, the driver that your operational system requires change it and you don't have it. So it won't work. Sometimes you require a kernel specification that you don't have so you have to recompile kernel or choose another line of version. So it's really a lot of things that could go wrong. And as I said, I have a certificate, I used TOS 1.1 and now I want to use 1.2. Well, you have to change something in your proxy or in your file. And that's a lot of things that you have to be aware of. She was just listing a few of the things that you need to keep track of but there are many more things. So, I have a customer who just ran some yum updates and it broke their (censored). I'm not going to say what customer it is, but they did a yum rollback, which is not supported, but they could not get this system up to save their lives. So now that the system's up support was like, "oh, we need the sos report." And he was like, "Bruh, I'm not reapplying these patches just to get you a sos report. And I gave you this log. I gave you that log. Y'all need to figure this out. I will not reapply. You have to tell me what broke it. I need to know what broke it." So a patch literally brought a production system down now to their credit they didn't... or discredit, they didn't test it, but that's a whole ‘nother story. Yeah. We're not going to go into that. So there's a lot of things to keep track of. Well, Vania and Ben, they've got a little bit of advice for us on how to avoid things going wrong. Nowadays nothing is really good for a long time. We encourage that every six months you have visit the release notes to see upgrades that are needed and to plan and put some time for your team to do the planning necessary to upgrade the systems, because it's not an easy task. You have to deal with system upgrade as you deal with a project of a software. So you have to have planning, you have to have timeline. You have to have people and you have to have, of course the material necessary for the upgrade. So the first thing she is suggesting that people do—that companies do—is to treat and update with the seriousness and the respect that it needs. Deal with it as you would any other software project that your company might be working on. Right? Allocate resources, put together a timeline, allocate engineering work to make sure that it gets rolled out properly. You need to plan for it like you would any other important task that your team needs to do. Now, part of that planning work is working with the affected teams, because it's not just your sysadmins that are rolling out these updates or upgrades by themselves. There are, especially for feature changes and stuff like that, you're going to be working with the teams that are integrating software into your system. This is a team effort. You are so right about this. You can't do it in a vacuum. And Ben had a little bit of advice on making sure that gets done properly. We worked early and often with them to notify that, "Hey, at some point we need to be planning around migration. Like we will need the apps that exist here to end up over here." And so that was part of the effort as well, is kind of preparing app teams for platform adoption, understanding their migration pain points. And, and what were some of the features we could build into the platform to help with that. It's not just your team. That's going to be affected by this, that a lot of teams are dependent on this as well. So, Johan, how do we make this easier on everyone? Well, lucky for us, Ben and Vania have a lot of experience with this and they've got some advice to share. Let's hear Vania's top recommendation for making updates a little less annoying. One of the things that will be crucial for your success is automation because automation, you diminish the human error and you guarantee the process is done equally all the time. So I'm really a defender of automation. I really enjoy automation all the process, because when you automate, you have like a contract of what you have to do, and it won't be able to forget, "oh, I forgot this step" no, it's automated. Let the automation to do this job. So, automating your testing process really makes sure that when you're rolling out an upgrade or an update that all the pieces of your software are still communicating with each other the way that they should. That you're not breaking anything with this update. It's something that a lot of people do manually, which sounds like a nightmare, especially for very large systems, but automating it really seems to make it much more of a breeze. You can imagine as you start to serialize and validate each and every kind of cluster that you offer, there's more and more kind of overhead for how you do that. So yeah, in terms of scaling, you definitely have to be on top of, I would say you have to be very good at automating the process because you certainly don't want to have a human button clicking and running scripts in order to execute and validate these kind of things. So anything you can do around automated validation is certainly helpful in this regard. So, you'll remember earlier in the episode, Vania recommended that companies upgrade their systems, update their systems every six months or so. Now for enterprise systems. These are big, they're complex. We've been talking about how there's a lot of moving parts that seems like quite a lot, quite frequent to do it every six months. But the best way to stay on top of that is to build out processes that systematize your updates. That means documenting the systems. Making sure that everything that needs to be checked is going to be checked. Building out that checklist and building out that automation and all of that takes a lot of time and effort, but you're front loading that work so that you can spend less time on each individual update without also worrying that update's the one that's going to be the one to break your production system. Now, she also has some advice that seemed a little counterintuitive because we've been talking about making sure that you get updates as soon as you can get them, but that's actually not something that she always suggests that you do. If you have a new bug or new issues, what advice to my customers is wait for the others to discover the problems. You don't have to do it on the first. Let's wait one month, two months and then we can upgrade your system. So she's talking about upgrades here, something that's not absolutely necessary like a security vulnerability or a bug fix or something that's affecting your user experience, do get those as soon as possible. But if it's an optional upgrade, go ahead and wait a little bit. Now, the final piece of advice that she has is to have like a developer or staging environment to test everything before it goes live so that if something does go wrong, it doesn't affect your business. They have this side environment, that's just a clone for production. They just clone everything that they have with difference because it's not a production really. It's not on the production so they can test all the integrations. That's the importance of having that test environment, but think about the cost that comes for having a duplicate of everything that needs to be updated. So there are actual costs and resources involved with that. So again, when we run up against those legacy systems that are sitting back there in the data center that don't have a clone, they don't have the ability to be cloned or duplicated and reproduced easily. We live in a time with containerization and virtualization. That really makes that easier, but that doesn't mean that there is still not costs involved with having these disparate environments to do this in. All right, Angela, Johan, let's come back to our question for this episode. How do we make updates, less annoying? Johan, I'm wondering, after talking with so many people about this topic, what have you learned? Updates are annoying, but there are things that you can do to make them easier and you want to make them easier because you want to make sure that you do them regularly? That if you keep them difficult and you give yourself a reason not to do them, then you're just going to end up in trouble. So it's really important to get those updates done for security reasons. It's really important to get them done for stability reasons and just to remain competitive in the overall tech landscape anyway. Because everything moves so quickly that if you leave your infrastructure behind to the point where you can't really upgrade it very much anymore, then you're leaving things on the table that could help you out in the long run. Angela, I'm curious, where's your head at, in all of this? I have a reinvigorated respect for people who have to go through these activities and these behaviors because we as—and I consider myself an end user at this point—we as end users don't understand just how much goes into it, how much planning, how much preparation to get it right, or as right as it could possibly be and the testing and the cross team communication. And there's so much that goes into that. I just have this respect because I was on that other side for a while, but listening to Vania and Ben, I really do have a greater appreciation for the things that go on that we don't see as end users and that we get kind of flustered by because, "This isn't working how I expect it. What the..." No one did it to you personally. Trust me when I tell you. So I think all of us as end users really needed to hear what they had to say. We wouldn't have the innovation that we have without folks like them. So listeners, after hearing what you just heard from our amazing guests, you have to tell us. What are some of your update, horror stories? I know if you're listening to Compiler, you probably have a couple of them up your sleeve and we want you to tell us all about them. So hit us up on Twitter or Instagram @RedHat always use the hashtag #CompilerPodcast. We really need to hear: What are some of your update horror stories? Let us know. And that does it for this episode of Compiler. Today's episode was produced by Johan Philippine and Caroline Craighead. Victoria Lawton pushes all of our buttons and makes sure we're all up to date. Our audio engineer is Elisabeth Hart. Special thanks to Shawn Cole. Our theme song was composed by Mary Ancheta. A big thank you to our guests, Ben Pritchett and Vania Vergara. Our audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Laura Barnes, Claire Allison, Nick Burns, Aaron Williamson, Karen King, Boo Boo Howse, Rachel Ertel, Mike Compton, Ocean Matthews and Laura Walters. If you like today's episode, please follow us, rate the show and even leave a review. It really does help the show. And we would love to see it, and hear from you. So thanks for listening. See you next time. All right.

More about automation and management

Automation reduces or eliminates repetitive tasks, typically in industries such as manufacturing and automotives. 

Articolo del blog

Uno sguardo all'approccio Policy as Code: perché ora e come può aiutare?

18 aprile 2024  |  

...

Automazione e gestione

Articolo del blog

Rivoluziona l'automazione dell'IT con la nuova integrazione di ServiceNow per Event-Driven Ansible

27 marzo 2024  |  

...

Automazione e gestione

About the show

Compiler

Do you want to stay on top of tech, but find you’re short on time? Compiler presents perspectives, topics, and insights from the industry—free from jargon and judgment. We want to discover where technology is headed beyond the headlines, and create a place for new IT professionals to learn, grow, and thrive. If you are enjoying the show, let us know, and use #CompilerPodcast to share our episodes.