ILM's Cary Phillips on the development of Cari for 'Dragonheart'

Speaker A: This podcast is sponsored by revision fx. Here's a quick name the company that made vfx for these movies. E.t. who framed roger rabbit, the abyss, terminator 2, forrest gump, the mask, twister, mission impossible, spawn, speed 2, saving private ryan, the perfect storm, AI, minority report, cowboys and aliens, lucy, skull island, the revenant, the batman, alien, romulus and running man. Thanks for inspiring us. Uh, ilm. A message from revisionfx. Find out more@revisioneffects.com. Hi everyone. Welcome to the Befores and Afters podcast. I'm Ian Fails from Befores and Afters. We're continuing our, uh, special ILM series with interviews I had done for the book industrial light and 50 years of innovation. Today we chat with Kerry Phillips, who is Director R, uh, and D Engineering Architect at ilm. Kerry has been recognised with three Academy Scientific and Technical Awards. These were for the Caricature animation system in 1998, the ILM creature dynamics system in 2001, and the ILM shape sculpting system in 2015. Caricature, or CARI as it was known, was famously used to help animate the face of Draco on Dragonheart. That's where I start the conversation with Kerry Phillips, asking him how Cary came to be. We also talk about the other SciTech awards and about the R&D environment at ILM, including the origins of Zeno. Let's jump into the chat now about caring.

Speaker B: Interesting. I've, uh, been thinking about this a little lately. Uh, last Thursday was my 30th anniversary at ILM. I started September 12th of 1994. We have a tradition of, on anniversaries, uh, writing up a little career summary and sending it out to the group. Kind of a trip down memory lane. When people ask me, like, what's your favorite show? There's a bunch of different ways to answer this, but certainly the, uh, overall experience, it has to be Dragon Heart. Just that kind of palpable sense of doing something that the world had never seen before. The first digital character, a, uh, starring role in a film, talking with a voice of an iconic boyce, Sean Connory. Um, it was such a dramatic, uh, leap forward at the time. And I mean I've, I've relayed this experience that, you know, uh, sitting in my office, minding my own business and, and Christian Ruay, the head of R and D, came by and said, uh, go talk to Alex Seiden. He wants to talk about facial animation on Dragon Heart and has some ideas. And he showed me a little mock up that he had Made of what an interface should look like, you know, the dragon's face and a, uh, timeline and a bunch of little postage stamp expressions with sliders beside him. Um, and it just kind of left off the screen of kind of immediately what he was after. And it was pretty clear it was all about interactivity. And this was in an era where everything in computer graphics was wading through molasses. You know, it was pipe into files, send it off to a process that would churn and come back with an image and then repeat that many times. And you'd get a bunch of images that you could send off to, you know, the abacus to be recorded on a videotape so that you could, you could play it back in in real time. Like the industry has come so far in terms of immediacy and feedback. And that was so foreign in the early days. And really what was so immediately obvious was that that what Alex and James Strauss, the animation director, had in mind was something that was interactive and immediate. So that clued me in to kind of the technical vision was to get the software out of the way. And there were various sort of systems at the time that were frameworks in which to do computer graphics, scene editing operations, but none of them were tuned for speed. And I set out kind of immediately just to, you know, based on my earlier experience from software I'd written in grad school and all, to write something that would just draw the Dragon's face as fast as I knew the hardware was capable of. Uh, so it was an exercise in optimization. This was before the era of GPUs and CUDA and all this kind of highly specialized, um, optimization. It wasn't that. It was just remove everything between the animators and the computer, make it as interactive as possible. And, um, that was what was startling about it at the time, was that you could grab a slider and slide it and the Dragon would smile and just kind of provided that immediate feedback. And I've often thought about how engineering is in a lot of cases, about getting things out of the way, getting technology out of the way. And that's really, you know, I just. The, the vision with caricature was to make something that did one thing but do it really, really, really well. And it lived for a long time because of, uh, that simple principle of, of, you know, it wasn't general, it wasn't extensible, you know, it wasn't flexible, you couldn't write plugins for it, you couldn't script it or do any of that kind of stuff. But it just did this one thing really, really, really, really well.

Speaker A: Um, the other two that you've received, and we'll talk about other things as well, Carrie, were related to Creature Dynamics and shape sculpting. Is there something you can say about those two together? And sort of the development of those over the years as well.

Speaker B: The Second one, the 2015 award, is that kind of more related to caricature? Let me get to that in a minute. But the um. So the Creature Dynamics Award was after Dragon Heart one of. There were a series of other shows that we worked on that involved one form or another of digital, uh, faces, digital characters doing. Sometimes just talking and sometimes doing odd kinds of things. Um, um, Flubber. Um, there was a shot in Flubber where Robin Williams grabs the flubber and stretches it over his face. And I remember that kind of getting the, the bending flubber to conform to the face was. That was a real challenge. There was another one, um, the sequence at the end of Harry Potter, one where Quirrell, ah, turns into Voldemort. Voldemort appears out the back of the head of, Of. Of Quirrell and um, this kind of uh. The back of a head that sort of morphed into a face. That was another one that kind of. There were um. Besides just you know, the dragon and Dragon Heart had. Draco had a vocabulary. He had certain things that his face could do. You can make it talk and all of that. But there were other. The sort of extension of that is. It's not a, it's not a vocabulary of phonemes. It's this weird kind of one off thing. And that kind of led us towards more algorithmic ways of doing that kind of thing. Our very, very first um, attempt at doing clothing was the capes on the aliens in um, uh, Mars Attacks. And that was so incredibly primitive that it barely qualifies as, as. As cloth simulation. But it kind of directed us in that um, that direction. And then it was really the ramp up to Phantom Menace that you know, there were a number of clothed characters. Jar Jar had a um, thing that he wore around his waist and he also had this thing, we call it his J. Crew sweater around his neck that kind of flapped and his ears sort of moved around. Boss Nass had these big jowls that shook and you know, there were a lot of uh, uh, bits and pieces that went together to make that film that sort of required a body to move in a way that animators didn't want to have to sculpt. Sort of um. You know frame by frame. And, and that was what led us to develop the physics simulation system that became our cloth flesh and hair simulation system used on a number of early films. And that's what was recognized, um, in the, the second SciTech award was, you know the, the dinosaurs in Jurassic park were really kind of the first creatures, but there was very little muscle flexing, jiggling. You know, in retrospect, looking back at those, the, the look of the characters was rich for the time, but by uh, modern standards, the movement of the skin was very, very primitive. And uh, I think that that second, the Dial M Creature Dynamics award kind of recognized the first time that physical simulation was brought to the animation process and added this sort of extra level of uh, physical believability and life breathed into the characters. So fast forward from that and that body of software that was caricature and the Creature dynamic System. Um, what I said before was it did one thing and did it really well. It actually did a couple things, the shape sculpting and uh, shape animation, um, as well. Technically, the award for caricature, the SciTech Award for caricature was specific to the dialing of face shapes. What came after that, but was not recognized by the award was for corrective shapes. The idea that you, through one process or another, whether it's hand animation and simple skinning or simulation process, you've got a character moving. Skin is moving in a way that the director has notes on. Either there's a fold in the skin that looks wrong, kind of inner penetration or something like that, or they just want a little more extra flex. And um, what we built in early on was an artist driven way of sculpting movements over time. There's lots of systems that, you know, Mudbox, zbrush, lots of cool modeling systems that allow an artist to work with something in ways that are suggestive of working with physical materials. But they tend to be all static in time. You know, you're looking at a static model and you're pushing the surround. We built this system in. The early version of caricature was the first implementation of it. The reimplementation in Zeno was what was recognized for the shape sculpting system. And really that focused on giving artists the ability to sculpt over time, to take something that was the surface that was moving through time and correct it, add details to it, uh, add artistry to it. And it became just an, and it still is to this day, something that we're kind of addicted to. I kind of, I've often made an analogy of like it's, it's a 3D equivalent of what Photoshop is to images. That you've got an image and it looks great. It's just got this one little thing in it. Go in and paint it out. Just paint it out and be uh, you know, you've got some blemish and you can go back and try to ask how was this generated and fix the process so that this doesn't, you know, this blemish doesn't occur. Or you can just go, go in, touch it up, fix it, boom, you're done. And that has been a layer that has for decades now been a major part of ILM's workflow of just plus it out by going in and very quickly, very easily touch it up on this frame, touch it up on that frame, blend between them, um, you know, just kind of dial things in and out over time. And that's really, you know, one philosophy of how you approach a system for representing organic anthropomorphic characters is build a physically plausible and mathematically correct modeling of what happens underneath the skin with bones and muscles and flesh and all of that and get it right. The other is sort of the artist driven technique of just take what you've got and make it look right, just fix it. One anecdote that uh, stood with me was um, you know, Thad Beyer. Yeah, friend of mine from college, from way back. He was actually the guy that got me interested in computer graphics. Uh, his passing was an enormous loss to everybody that he, he touched. Um, we went to college together, we worked at PDI together, came to ilm. He was on the um, the committee that um, that investigated and gave the award to Caricature. And the year after that the PDI animation system was investigated. The PDI facial animation system was investigated. And Thad told me that it was kind of the, the difference between these two systems was rather stark because the system at PDI was very heavy, very structured, you know, uh, a, uh, template around the definition of a face and how it moves. And it was hard, it was difficult to set up and all of that. And they do basically cartoons with it. The system at ilm where we do hyper photorealistic character, you know, animals and, and, and people, was very direct, immediate, interactive and artist driven. It's kind of the opposite of what you might expect, but it's, it's kind of an interesting, you know, anecdote or pair of anecdotes about the differences in philosophies that you might bring to These kinds of problems. But my perspective on this was always just kind of give the artist what the artist needs to just do their work as immediately as possible. And what, what I kind of

Speaker A: uh,

Speaker B: value the most about that early experience was um, um, feeling like I'm empowering these people that are not sophisticated computer, you know, hyper technical computer users to work their magic. Some of the feedback that I got, comments that I got that have stood with me uh, the most over the years are the non technical animators on Dragon Heart who said, you know, I, I didn't realize I had the ability to do this. And you know, it's because at the time there was no, there was no other way to, there was no other systems, no other place that was doing this kind of thing. So the kind uh, of giving, giving an artist the ability to create something that they didn't know they had the ability to create was just a really, really special experience.

Speaker A: What about my other favorite thing I'd like to talk to you about, which is Zeno, which I think is a really important part of ILM and ILM's history. Tell me a bit about building up Zeno over the years as well.

Speaker B: So I think one of the original motivations for developing Zeno from the very beginning was rooted in what I just said about caricature, that it was a system that was designed to do one thing and do it really well. It was not engineered from the ground up to be flexible, extensible, you know, configurable. There was no plugin interface. So early on in the development of technology for Episode one, Florian Kynes, who has done an amazing spectrum of stuff over the years, um, was given the challenge of developing a system for managing the terrain in the pod race on episode one. So you've got this massive expanse of sort of desert and um, mesas and all that. You're flying by at ah, 500 miles an hour, five feet off the ground. So you've got massive amount of geometry that you're moving through. How do we manage that? And so, okay, we need a system for that. And he looked at the framework that character represented and pretty much immediately concluded this is not up to the job. And he was right. It wasn't designed for that. So it wasn't, it wasn't a sort of framework that was up to the task of being extended to work in different, for different applications. And this was before the era of Maya, so using Maya wasn't an option. Um, so that set Florian on the path of developing a core data representation. This is how we represent geometry that moves over time, that we can animate and um, structure different kinds of data that interact to go into, to make um, the content of a shot. Florian was paired with Jim Horahan who had experience developing interactive systems both uh, you know, dynamic and then at ilm, the interactive component of um, you know Isculpt and caricature and the pair of kind of Florian on the core representation um, of the scene and Jim on the application structure created this system that eventually became Xeno. And the origin sort of came out of episode one. The original motivation, the original uh, thing was the pod race. And it didn't turn out that that technology ended up getting used on the film but it did get used on the in battle sequence. The droid slicing um, uh, in the final battle um, was done with this system that they came up with it. It was called the Choreographer back then and it was used to uh, to, to do the, the simulation of the, the droids being sliced up. And it was also used to choreograph the, the Gungans and the, the droids in the, the final battle. After that, after the conclusion of episode one, that body of software kind of became a solution in search of a problem. We had this thing, it was designed to be you know a solid framework that we could extend in various different ways. But uh, what are we, what are we doing with it? And we did several things with it. We extended the rigid body simulation system in it and it was used on Pearl harbor. The crashing planes in Pearl Harbor. Steve Sullivan joined ILM as a computer vision expert and took that framework and built out a camera match move system and dragged the layout or the match movers into the modern era of computer vision based motion and camera tracking and that sort of, you know, Xeno. The body of software that became known as Zeno didn't start out to be that. But that's really the thing that kind of put it on the map. Early on there was, I mean it, it had been called Choreographer. It had also been called Fred for Bread, Astaire and ah, at one point I remember we were in a meeting trying to decide what are we going to do with this? And we, somebody kind of mused that it always felt halfway done. And somebody joked like yeah, it's like Zeno's Paradox. The arrow is always, you know, it goes halfway to the wall and then halfway, it's always halfway there. So we kind of joked, yeah, let's call it Xeno because it's always halfway done. The name kind of stuck so um, you know it, it became this sort of general framework with certain features that were built out to be really powerful. It also kind of sort of the trajectory ended up paralleling Maya. Uh, it kind of came to production right about the time that Maya came out a little bit, it was a little bit before it. So we ILM always kind of a bit struggled with. We have these two systems that are both very powerful. Zeno had some features that were just really killer. The motion tracking uh, and all especially was super powerful really early on, kind of rather intoxicating while the animation components of Maya uh, really came into their own. So some most other studios, I think that came along a little bit later sort of bought into from the outset that Maya is the framework and we build into that. ILM was a little bit late to that because we had Zeno and it provided a similar kind of structure. It had an early leg up on some of the power features but it never got the attention to usability and polish that it could have. I've also kind of over the years compared it to Presto at ah, Pixar and I think Pixar excelled at designing an interface that really presents the information to animators and artists in an accessible way. Zeno never got that early love. It was always sort of build in the power features be driven by the super technical senior artists pushing the boundaries of what's possible. And it always has kind of had um, the quality of the system in which all the wires are exposed and you know it, it's not easy to use, it's not easy to learn. But once you learn it, once you, you, you become the master of the, of the, the power, it's kind of intoxicating and, and a lot of artists have gotten a tremendous amount of mileage out of it. There's a lot of artists at ILM that will um, will, will swear about how, how great and powerful it is. But it, the rap against it has always been again, it's hard to use, it's hard to learn and it's non standard. So you know, if you talk to people about ilm, uh, about Zeno at ILM over the years, you're going to hear a lot of different stories across a very wide spectrum. Again, there's, there's a core group of people who have always deeply appreciated its power and there's others that have always been suspicious of it and felt like other tools are, are better suited. And along the way many of the special purpose workflows that we built into Zeno have gradually been deprecated in favor of other tools. So, um, um, we built out a lighting editor in Zeno and used it for a few years. And then Katana came along and we shut that down and started using Katana. Same with effect simulation in Houdini. Um, Vue Paint was. There was an early version of viewpaint and then a reimplementation of it in Zeno, which we used for many years. And now we use Mari and Substance. And so there's a number of things that Zeno kind of pioneered for the super technical, um, uh, power users early on that eventually kind of the calculus sort of changed to favor the more standard tool, um, that people know that's easier to learn, that's easier for people to come in and adopt. There's still a number of things that Zeno really, really excels at that we still really rely on it, but. But, um, it's not nearly as much as it used to be, but it's also 25 years old. So for where package to last for that long, um, is kind of remarkable.

Speaker A: Just to finish up, uh, this is sort of more of a general question about R and D and innovation at ilm. I wonder if you can say anything about how you feel the culture of ILM enables that innovation to keep happening because it really has over, uh, 50 years, you know, almost 50 years.

Speaker B: Let me tell you a story that I often tell people. It's a story about Joe Leteri. You know, Joe was at ilm, who I worked with Joe early on, and he left for Weta. What I've always understood about Weta, uh, is that the way that Weta works, the techniques that WTA uses, uh, to a large degree emanate from Joe's knowledge and experience and wisdom. Like the way things work there are kind of the way Joe thinks they ought to work. And having that kind of brilliant, masterful overall vision for how things should work is one way that a studio can work. ILM does not work that way. There's nobody at ILM that occupies that kind of technical vision or we have absolutely brilliant, technically brilliant and creatively brilliant people. John Knoll is one of them. There are many others of, you know, people that, that have, um, brilliant ideas, but they're not the kind of overriding sort of directives of this is broadly how everything should work. ILM is the hive mind. And, you know, there is a saying around here that, to me, it is attributed to John Noel. A good idea is a good idea, regardless of where it comes from. We very much have the spirit that um, we trust that everybody has experience and ideas, and we want to hear them. So the conversation, the discourse at ilm, um, is a lot of really smart people saying what they think. The challenge is how do you filter through that? Um, we have trust that everybody approaches that in the right spirit and the good ideas win out in the, in the discussion. So we have trust that people will drive towards consensus. Sometimes that works better than others. But, um, but, but it is very much the value of your idea stands on the value of your idea, not your assigned title. And, um, so it very much is a place where everybody is expected to contribute ideas. And there's always been an openness of, I'm, um, facing a problem. What do I do? I ask around, I ask for help. I, I bring in people who can help me jog, you know, my me out of my, my rut and, uh, and figure out, you know, what else, what else do we do here. So it is very much the hive mind of lots of really smart, creative people who, um, come together to. ILM has never impressed me as a particularly political place. There have been a lot of long, simmering arguments about how things ought to work, but they're always rooted in what is the right way to do, what's the right thing to do here, what's the best, what's the best approach. People can bring very strongly held convictions about that, but they're bringing a real belief in a particular solution or a particular understanding of a problem. They're not bringing this sort of, I want this to be done my way for, you know, my own sort of benefit. That's, that's been my experience.

Speaker A: And I'm so glad you summarize it like that, Carrie. That's such a great way to end on. Always fun to get the Carey stories. And I'm, uh, sorry to make you repeat it so many times, but, uh, really appreciate it.

Speaker B: It's, uh, fun to relive the old stories. Thank you.

Speaker A: This podcast is sponsored by revision fx. Here's a quick quiz. Name the company that made vfx for these movies. E.t. who framed roger rabbit, the abyss, terminator 2, forrest gump, the mask, twister, mission impossible, spawn, speed 2, saving private ryan, the perfect storm, AI, minority report, cowboys and aliens, lucy, skull island, the revenant, the batman, alien, romulus and running man. Thanks for inspiring us. Ilm a message from revisionfx. Find out more@revisioneffects.com.