How Does Data Help Shape Movies?

  |  Compiler Team  
IA/ML

Compiler • • How Does Data Help Shape Movies? | Compiler

How Does Data Help Shape Movies? | Compiler

About the episode

Movies are culturally important. They transform language and communication. Motion pictures present fantasy worlds we can get lost in, helping us understand the world differently. Discussing data and movies can make the fantasy seem…a little less fantastic. It can feel sterile, mass produced, and devoid of imagination.

But data is vital, both for those behind the camera and those sitting in theaters (or at home).  This episode will cover some ways data science and machine learning can inform filmmaking, from conception to post-production.

Compiler team Red Hat original show

Suscribir

Subscribe here:

Listen on Apple Podcasts Listen on Spotify Subscribe via RSS Feed

Transcripción

Angela, Brent - let's play a game. Oh boy. I'm game! I'm going to read you a quote from a movie and you're going to guess the movie. I love this game. I can already tell you, I'm going to lose. Okay, okay. There's no winners or losers here. I just want to illustrate a point. Okay. Ready? Yes. Okay. "I'm going to make him an offer he can't refuse." The Godfather. The Godfather. All right. Good. Okay. Let's try: "There's no place like home." Dorothy in the Wizard of Oz. Yep. "Go ahead, make my day." The Terminator. I was biting my tongue but- Was that right? No, it was not right, Brent. It wasn't right? No. Dirty Harry. Yes. Dirty Harry, Clint Eastwood. Wait, that wasn't Terminator? No. No. Why did I think that was Terminator? It's Terminator-esque. Same bravado, I guess. I don't know. Here's one: "Hasta la vista, baby." There that's yours, Brent. Oh, Terminator. There we go. That was the Terminator? Yes. Okay. Yes, it was. So do you know why I wanted to play this little game? I actually have no idea. But I loved it. It's my favorite time of the year. It's summer movie season. Oh yeah. It's the most wonderful time of the year. Yes. Especially for someone like me. I love movies. I watch movies all the time. They have important meaning to me. For many people, they have a lot of significance. A lot of cultural significance, even. Movies can transcend different cultures and different languages, even. There's so much to be said about how film connects us all together. Connects us all and also helps us understand ourselves. Yes. It's a powerful tool. Very. So, I wanted to talk today about data science and filmmaking. I mean, that's interesting because I don't typically think of data science and movies together. I understand that and it even feels sterile or lacking in creativity to talk about data and film but I'm happy to tell you that there are some amazing ways that data science can inform the movies we watch. Really? Okay. Well, what do you have for us? Let's do it. This is Compiler, an original podcast from Red Hat. We are your hosts. I'm Brent Simoneaux. And I'm Angela Andrews. We're here to break down questions from the tech industry, big, small, and well, sometimes strange. Each episode, we go out in search of answers from red hatters and people they're connected to. In today's episode, how does data science shape what we see in movies? Producer Kim Huang is here to help us out. First, I spoke with Madeline Di Nonno, president and CEO of the Geena Davis Institute on Gender in Media. She was kind enough to speak with me and she has a lot of experience with data science and filmmaking. She says, "Data science is important in the institute's work." Data has always been the key for us in terms of raising awareness, identifying unconscious bias in content, and also using it for advocacy. We are a data driven advocacy group and we have found that has been the best way for us to have a dialogue in a way that is not defensive. Having the data, it's the facts, it's not opinion, it's not theory, and it has really helped us not only make the case, the social imperative, the business imperative, but also has allowed us to measure progress. Wait, Kim. Tell us about the institute. The institute was founded in the early 2000s by actress Geena Davis. I'm sure a lot of people know who she is. Oh yeah. Yep. They conduct studies in research to adjust the biases and what we see on screen and foster more inclusion in media. Oh. So they're really interested in representation on the screen. And so, they're trying to do this in a quantitative way so they're having people watch these films and then tally up? Yep. Like who's on screen and how much they're speaking, is that right? Yes, exactly. That's interesting. This does sound like this is prime for some sort of data science because the way that you just explained it, it really says we're looking at data points. Right. And how do we add those data points up? To your point, when humans are doing it, that has its own bias but let's be clear, machines are built by humans. That's true. Yeah. That does also, in turn, there is bias inherently in those as well. Indeed. Yes, and we're going to talk a little bit about that later on because I'm glad you brought that up. In 2013, the Geena Davis Institute was approached by Google. Together, they wanted to see if machine learning could help them dig deeper in research. Together we developed what became GD-IQ which, hmm, what does the GD stand for out there? It's Geena Davis Inclusion Quotient and essentially what it allowed us to do is identify not only how many female characters there were on screen but their screen and speaking time. That was something that could not be measured by humans accurately. GD-IQ is a hybrid methodology so it combines machine learning and also, we have a team of human expert coders that provide the other dimensions that we look at in terms of race ethnicity, LGBTQIA, disabilities, age 50 plus, and body type. So what did they find out? It's one thing, right? If you have a bunch of people or even an algorithm there to process how many people are on screen at one time or how many people are speaking or who they're speaking to even, right? What you want is a robust picture so you have human expert analysts alongside a sophisticated algorithm that was built specifically to count speaking time, amount of being seen on camera for your face and not just your body for example, the different people who are speaking, how old are they, what do they look like, how are they dressed. When we looked at female characters from 2016 to 2020, we saw that female character screen time increased by 8.4% and female character speaking time increased by 7%. It's one thing to count how many female characters there are. It's another thing when you're looking at a screen, it's like, do they have the same presence as their male counterparts? And all of that has been possible because of having the GD-IQ. The introduction of data science and machine learning for the Geena Davis Institute strengthened the case that so many people have made for decades, movies have a lot of problems with representation. It's interesting that this institute took this on because since the beginning of movie making, when characters weren't even played by the same race or the same gender, there was so much misrepresentation in movies. Now, we've come to 2022 and we're hearing that, "Yes, it is making improvements and those improvements are happening very incrementally." We all see it. We all know it. Every TV show that we turn on and especially if you're in the minority, you really feel it because you never see yourself, you don't hear yourself, but you're always consuming this content that is never representative of who you are. Yeah. That is so true. I mean, I remember some of the first times that I saw a queer person on screen. I don't know. I can't tell you how transformative that was for me. I believe it. I believe it. Yeah, totally. I'm speaking for myself and probably speaking for Kim, those things happen to us as well. Yes. Well, like she was saying earlier, having this data helps us have, I think, conversations that are grounded in fact and not just assertions because so often, these types of conversations can get overheated in some ways. For sure. Yeah. Yes. And they turn out to be not productive. Exactly. Kim, help us out here though. We're talking about how data science can help in the creation of films but it sounds like the work that they're doing is analyzing films that already exist. What's the connection there? Yeah. That's a really good point. I was also curious and it turns out that Madeline had a lot to say about that. When we started socializing GD-IQ back in 2014 to 2015, it works on audio and video and it's a great product but it's been used more as an auditing tool. It's not something that is being used for the most part during the creative process and it's a great way to measure change. We were really looking for an intervention. How can we be preventative? How can we help our partners be able to assess content before they moved into the creative process of casting, of producing. Our partners, led by Dr. Shri Narayanan at the SAIL Laboratory USC Viterbi, have a patented text tool IP. Geena and I approached them and said, "We want to create Spellcheck for Bias." What is Spellcheck for Bias? Spellcheck for Bias is a combined methodology where it is led by our research department and also incorporates some machine learning into it and it's a way for us to look at who is showing up and how are they showing up. For the most part, most writers don't describe every single character that is contributing to dialogue, that's just Script Writing 101. This allows us to examine all the characters that are contributing dialogue in a show, in a movie. It's a way for us to organize them and also walk them through these six dimension tests. We look at gender, we look at race ethnicity, we look at disability, we look at age, we look at body type, we look at LGBTQIA, and it's a way to very quickly assess the type of characters that are in a script and if they're organized or described. It allows the creative people to get an easy assessment and then decide what they would like to do without invading the storyline, without invading the authentic truth of the story, and the storyteller. This Spellcheck for Bias is actually an application that runs through a script… Yes. And it uses the dimension tests to look at the characters in all of the biases that could be in there. I just think that's pretty interesting. Like she said in here, "We're not going to go in here and mess with the story but we're looking at the script and we're saying, "Okay, this is what you have." Does this look right to you? I mean, I guess that's probably the question because it's an audit and it's not very preventative. Yeah. If you do it this way, it becomes a more preventative process and it's like, "Oh, we're skewing too far the other way." It could be totally unintentional but until you bring these issues up, again and again, they tend to get overlooked. So this is cool that you get to know these things before you start shooting and you can fix them. Yeah, she's right. So many times, tools like this are used for auditing purposes and I love that flip to say how can they also use them for generative purposes. Like how can they help us create better work, not just assess work better? Yes, exactly. Madeline says the ultimate goal the institute has with all of the work that it's doing is to show equal representation on screen that's comparable with real life populations. But if I could play devil's advocate for a second, movies aren't real. Why do we have to see on screen what we see in real life? People like Madeline believe that what we talked about earlier on the show is true: Movies have a significant impact on culture. And a fantasy world is an escape but it also has only a few personal stakes, if that, for a person who's watching it. Challenging our biases within that environment could be easier than trying to do so in real life. We're talking about fiction. We're talking about the world of make believe. Clearly, unless there is something factual from history, there's no reason why in the 21st century, we shouldn't be a parity now. I mean, parity should be the low hanging fruit, not even something we want to ascend to. Our goal is to really, at least, achieve parity and then go beyond it. Hopefully, we'd love to be out of business. That is the ultimate goal. Why do you think the film industry is taking inclusion so seriously? Why now? Well, Madeline says it's just as much about business sense as it is about common sense. There's the social imperative which we've been talking to a lot but there is a commercial and business imperative. We have found that when you have diverse content that has leads and co-leads that are representative of BIPOC and many other groups, it will make more money. Put my people on screen, we can make you money. Exactly. Exactly, and it's not just conjecture either. A study that the institute conducted in 2015 examined the highest grossing movies of that year. They found that movies with female leads grossed 15.8% more than ones with male and movies with both male and female leads grossed 23.5% more than movies with just one or the other gender in the lead. Wow. That is the pockets right there. That's literally proof positive. This matters. Yeah. There are tons of papers and reports that I could cite but they all paint a similar picture. Data science is helping to make that picture more vivid and more actionable but there are ways that data can inform the production process as well. I wanted to know more about how data can affect the technology of modern cinema. This time, I wanted to know more about data science in the production process. After a long search and a lot of emailing people, I found James Blevins. He works in post production on a number of different shows and projects. The majority of which he could not tell me about because of NDA, but just to give you an idea about the type of projects that he works on… Most recently, I was the post production supervisor on a show called The Mandalorian and that's the latest feather in the cap. What? Yes. Okay. Okay. What is Mandalorian? Oh gosh. It is everything, Brent. It is everything. Brent, it is a Star War. Oh, it's a Star War. Okay. With quite possibly the cutest fictional character in recent memory. Oh! But before he could bring Baby Yoda into our homes and our hearts- Oh, that's where Baby Yoda comes from. Yes. Grogu. James spent four and a half years at Netflix. Okay. Just as it was beginning to roll out its original programming. But in the short time that James was at Netflix, the company scaled its original programming from 8 shows to 600- Wow. Properties. Word? A year. Yeah. They went from Orange Is The New Black and House of Cards to all kinds of stuff in no time flat. Yes. Wow. We really shattered our own expectations. I think after about my third year there, I started changing the way I approached my work by saying, "Well, 600 originals. You know what? There are people who work on 600 things a year all over the place. It's really just 600 things." You just need to wrap your head around that this is really not that difficult if you can just move your own internal decimal point and start thinking about these things. It's not, "Oh my god, it's 600 things." It's, "It's only 600 things." Oh. Oh, just move your decimal point. Okay. Is that how we're doing that now? Got you. What a mindset. I wish… on my best day. How do you move from 6 to 60 to 600? How does that scale happen? Yeah. Good question. When I asked James about how Netflix uses data to scale up their production to that extent, he was very tight-lipped. It's kind of the secret sauce of Netflix and other streaming services as well, right? So he couldn't say very much but he did illustrate something really important about how Netflix realized that the power of transforming its service lied in its own data. The data that it had, that was being generated by its own customers at the time and how valuable that was. I think what became really clear was the differentiator between what was widely understood and published information. And then, there was the data that you owned yourself and one could become one's own Nielsen and that was amazing to watch and how it affected deal making. To be clear, he's talking about Netflix becoming their own Nielsen rating service. Meaning they have the data of what their customers are watching and they can use that to scale and see where the wind is blowing. Yeah. Right. That's kind of become, I think, a business model for a lot of streaming services since then but Netflix was the first in that space. Data is king. Yeah. I mean, you're sitting on a whole mine of first party data. You have a lot of that data already. What do you need Nielsen for? Right. Exactly. Exactly. When we think of big budget films, the ones that are coming to theaters and streaming services in the next few months, it's easy to think of big workstations, high powered computers that render special effects. James says that all of that is about to change. We really shouldn't care about whether or not we have a fast enough box on our desktop. We should just wonder whether or not we have access to compute, right? And know how much compute we need and... There's a whole economy around all of that, that's happening right now. Well, I think when you're talking data, you're talking about compute. Bingo. Yes. And where does that compute increasingly finding itself, Angela? In the cloud. That's right. That's why we're not worrying about those big old clunky workstations and everyone had to do their rendering. No; it's a service now. Just offload it. That's right. And you still keep moving forward so you're not bogged down. I love how we're tying all of these little tech threads together. I love how we do this in this show. Oh yeah. I mean, I think about the transition from physical film to digital filmmaking and then I guess there's a lot higher quality of filmmaking now in terms of like 4K and HD. Some of the issues with filmmaking do have relationships with translating them into high-def, HD, and then from HD to 4K movies. Especially ones made using an older technology were simply not made to be seen on our very new, very crisp screens that we look at movies with now at home. So in post production, people like James use machine learning to scale up visuals. So, I use those types of things like taking material that was at HD and bringing it up to 4K standards using those machine learning algorithms, sort of a no brainer. You are also looking at a world where all of your televisions are about to get really bright and really big. Turns out that the DPs of the world, when they were shooting, they did not expect this to be where their show was going to be presented to the audience. Their hero was a Dolby Vision theater, right? But even those Dolby Vision theaters are beginning to run things at 48 frames per second. I don't think I understand those. Okay. Kim, can you help me out? Yeah. You're going to have to bring this back down. Sure. DPs are directors of photography, what we know as cinematographers. In film, they're responsible for making sure that the visuals are exactly what the director is looking for, that everything on screen looks the way it's supposed to look. I think what he's saying is that films that were made previously, they're meant to be shown in a theater on a screen but that's not necessarily how we're watching these movies anymore. Right. I'll let James explain and I'll come in a little bit after to put some context down. But a lot of that has to do with: how frame rate shows up in a movie theater is very different than how it shows up on your screen at home. The reason is, because the manufacturers don't want people to look at the movies, the great premier pieces of content today, and see all of this judder. When you bring the contrast up and brightness up, you'll begin to see things that you never saw before and will make the DP squirm which is this latent imaging that your brain can't quite get rid of the old image and bring in the new. When you're looking at something really bright and high contrast, it's burned into the back of your ocular system and so you'll begin to see this uncomfortable judder. Uncomfortable judder. Yes. Sometimes in older films on HD screens, you'll see jittery outlines around shapes and moving objects. That's what he means when he says judder. Oh, okay. So, how else are they using data science? Data science already has a place in helping production technology keep pace with advancements in screen resolution, for example. What else? We had this unpleasant experience in some of the more premier products that people tried to watch things running at very high frame rates and it looked too video-ish, right? There was that weird overly clear soap opera effect. Well, it turns out that if you compute the edge of every object in something with a lot of samples, say something that was shot up 48 frame per second or even 120 frame per second, if you analyze that material then you can then choose what your shutter angle is going to be in post and add the proper amount of blur that is to your satisfaction. And so, some of this work's being done today and that will result in probably a master at 48 frames per second. And so, all of us are preparing for a world where creatives turn to the streamers of the world and say, "Hey, I thought you were filmmaker friendly." The theaters are showing our stuff at 48 fps and we made a scene by scene calculation about how we wanted our motion grade to look in post. What about you guys doing that for us? And so, I think that conversation is about to happen. That will require quite a bit of compute and quite a bit of, I mean, nothing insurmountable, a day's worth of compute on some... Right now those solutions are in boxes in rooms but those could easily be up in the cloud. So do you want me to break this down a little bit? Please. Yeah. I'm going to need that. All right. What James is saying when he's talking about the compute or he was talking about doing scene by scene calculations: streaming services, Hulu, Netflix, Apple TV, etc., they can present films and TV shows running at too high of a frame rate while the industry standard is much lower and gives that cinematic visual to it that people are used to and that directors and creators want. Machine learning and data science can do scene by scene calculations and help render these films at that industry standard for streaming services without too much human intervention i.e., a lot of post production work. That's how people get to see the movies that we want to see on these devices in a way that we're supposed to see them. Kim, let's pull this all together. Our question for today's episode was "how does data science shape what we see in movies," so what's the answer? It already does shape a lot of what we see on screen in a lot of ways that are unexpected and are actually really exciting. I feel like the attitude of data being in film or being a part of the creative process makes people hesitate a bit. There's a lot of sources of data, of information for people to use more than ever before especially if you're making movies. It can be generated by other platforms like social media platforms or you can even buy from vendors like Nielsen. It can be captured at the source if you have your own streaming platform like streaming services that we have now that use complex datasets to present their customers with what they want when they want it. Also, it takes a lot of work to bring these films to life only for the experience of seeing them to become inconsistent if we want to, say, stream it at home or watch it on that brand new 4K TV. Data science can help us give those cinematic moments back to viewers and help cinematographers and directors present their exact vision agnostic of where or how we're watching it. Integrating data science does not have to feel sterile. It can empower independent creators making things for undiscovered audiences. It can help visually present movies in line with the creator's vision. It can also help curb biases of which stories get created and which go untold and it can help address those biases that affect who we see on camera and who is behind the camera as well. Those are all good things. Indeed. I love how this episode ties in with so many of our previous episodes and it's amazing to see just the advances that data science and technology as a whole is doing for the film industry. It's only making it better and more inclusive. That does it for this episode of Compiler. Today's episode was produced by Kim Huang and Caroline Creaghead. Victoria Lawton always tells us to do good because doing good is good business. Mm-hmm (affirmative). Our audio engineer is Kristie Chan. Special thanks to Shawn Cole. Our theme song was composed by Mary Ancheta A big thank you to our guests Madeline Di Nonno and James Blevins. Our audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Laura Barnes, Claire Allison, Nick Burns, Aaron Williamson, Karen King, Boo Boo Howse, Rachel Ertel, Mike Compton, Ocean Matthews, and Laura Walters. If you like today's episode, please follow us, rate the show, and leave a review. It really does help us out. We love that you listen and keep on listening. Bye, everybody. All right.

More about artificial intelligence

Usually referring to processes and algorithms able to use data to simulate human intelligence, artificial intelligence can provide insights based on acquired knowledge.

Publicaciones en el blog

Pensar en grande, comenzar de a poco: por qué la inteligencia artificial enfocada triunfará en 2025

4 de marzo de 2025  |  

...

IA/ML

Publicaciones en el blog

Sugerencias para los patrones de peticiones del modelo de lenguaje de gran tamaño (LLM) de IA generativa

1 de julio de 2024  |  

...

IA/ML

Programas originales

Steve Wozniak | Command Line Heroes

Historia de la tecnología

Programas originales

Command Line Heroes: Cuarta temporada: Hardware open source: la alianza de los fabricantes

Open source
Historia de la tecnología

Sobre el podcast

Compiler

Do you want to stay on top of tech, but find you’re short on time? Compiler presents perspectives, topics, and insights from the industry—free from jargon and judgment. We want to discover where technology is headed beyond the headlines, and create a place for new IT professionals to learn, grow, and thrive. If you are enjoying the show, let us know, and use #CompilerPodcast to share our episodes.