“Still to this day, there’s no commercial software fully able to program these things. We wrote programs that would allow us to model and generate the files in that way. I was very lucky to be a part of that,” Nick says. “It was around that time that a classmate of mine came down with a brain tumor. His name was Steven Keating, and he was an MIT student.
At the end of this journey into the possibilities of Voxel Printing technology, the question that comes to mind is whether it could ever be used for producing functional organs. While bioprinting of complex organs is infinitely more challenging than what many outside the medical segment perceive, this complexity ties back to what Nick discussed about the lack of exact boundaries in human tissue.
Rob’s team connected with the team at Fraunhofer who already developing a Slicer to take advantage of the J750 full-color printer that was coming out. They worked together to develop their Slicing tool, which was a way of taking a texture-mapped object and turning it into these bitmaps that you could then print in full color.
While Rob and his team needed to design multi-material objects, medical imaging is inherently a full volume of multi-material densities. “I thought, ‘Wow, that’s going to be cool’. We started to see if we could take the data that he was working with and reproduce it as a 3D print,” Rob says.
“We have a lot of abstract ideas about things that we might try to do, but without a concrete application, it’s hard to drive some of that development. So the studio benefits from having something to push and pull on these tools that we’re trying to create. And I’ve been feeding that back on the current production we’re doing right now because we’re are working to generate a lot of things where the ability to author the full volume of the object is going to be critical. That’s the direct benefit,” Rob says, highlighting the “personal benefit of just expanding my worldview and my knowledge in this area. Nick brings these cases in front of me and it’s fascinating.”
Epilepsy is not a condition that anyone can see, like a tumor or a broken bone. It’s the result of multiple factors. The best way to identify where it is is by using physiological rather than morphological data. “You’ve got to hone in on it by overlaying all these forms of data,” Nick explains. “The problem is there’s no single solution to do this. Planning surgery involves looking at many different data sequentially, on two-dimensional screens. It’s a massive mental exercise for the team to be able to reconstruct all this data
“It was at AMUG in 2015 – Rob remembers. “I sat down next to Todd Grim. He asked me what we do and I explained. He said that we should speak with these guys at Fraunhofer in Germany, who had figured out how to add a fourth color to the three-color printer by dying the support. And had created a Slicer to go with it”.
Laika used Voxel Printing on a J750 3D printer to bring the main character from the recent Missing Link to life. (Photo: LAIKA)
Building bitmap bodies
Nick’s background is quite diverse. “I got started in 3D printing in a bizarre way,” he tells 3dpbm. “I started in architecture. In my first year at architecture school, we had drafting boards. In my second year, I watched them rip out the drafting boards and give us computers. Throughout my education as an architect, I was continually exposed to this swap from 2D to 3D. It was like a series of epiphanies. Even with simple technology like SketchUp we could now make a building in 3D and take it off the two-dimensional plane. It was a revelation. I could understand it better, and convey it better. Explore new opportunities in design….
Nicholas Jacobson, Translational Research Faculty at the University of Colorado Anschutz Medical Campus
Now, the next dimension of technology that is coming into the medical world is focused on physiology. This new data form, which is usually vector-based, is not something that’s seen in the body but is derived from other forms of medical imaging. For example, one of the things that Nick’s team invented was the ability to see blood flow and velocity from an MRI scan, which is called 4D flow.
Additional developments of AM technology could see the introduction of new elements, such as moving parts or flowing liquids, as well as the ability to mimic a virtually infinite gradient of densities and color. Work is ongoing to develop new materials and on representing the body through durometer or density mapping. In addition, while the brain may be the most complex possible application of Voxel Printing, other areas can also involve models of the heart, cranial and facial reconstructions, liver, and kidneys as well as research in transgender and vasculature applications.
“What we’re doing is we are taking all that data, with its own set of problems and requirements, translate and fuse that to present a single holistic model, Nick clarifies. “Everyone understands things in 3D, so the huge team involved will be able to – hopefully – come to a consensus more easily. And guide the surgeon before and during surgery.
Keating teaches a class of first-year students at MIT.
From bitmap to full-color Voxel Printing
At the AMUG conference, Rob and his team would often try to find other people who were using similar techniques. They had access to an advanced feature of PolyJet printing, which was called bitmap printing at the time, and has now been rebranded as Voxel Printing. They had been to university campuses and saw research teams that were playing around with these machines and using bitmap printing. One year at AMUG Rob listened to Nick’s talk about his work in the medical field.
Some of these challenges consisted in recreating something like human physiology that has no hard physical boundaries and is in a constant state of change.
LAIKA is also exploring how to animate a character as if it’s a real living breathing thing. “In the most straightforward way, we create miniature-sized people that must function like a person, Rob says. “On top of that, it’s also a conceptual framework around which I can refine these tool sets for designing in full volumetric voxels.
PolyJet’s multi-color capabilities first came out in a three-material printer. Rob worked with researchers at Virginia Tech and generated a toolset to do four characters on Kubo and the Two Strings that look like full-color plastic prints, but they only used three materials. They were intermixing by generating slices and setting the slices as bitmaps to the printer. This enabled them to achieve a more dynamic look on these printed characters that didn’t have to be painted over the top, which can be seen Coraline. Then the full 7-material [6-material plus support] machines came out.
“We also see a lot of surgeons with a high level of technological sophistication that want to be able to design their own tools. I’m hoping that what Rob and I are doing is expanding the range of opportunities for 3D printing. Very few people are looking at soft tissue or have the ability to do that, and soft tissue makes up the majority of our body.”
Voxel bioprinting
Brain FX
Before getting into the details of this inspiring story, a few words on Bitmap printing. This term, which is now known as Voxel Printing, refers to the ability to control material at the voxel level during a print. While several AM technology providers offer some level of voxel control, the company that has made the most progress is Stratasys via its PolyJet technology (which was originally developed by Objet and acquired during the 2012 merger). Voxel Printing allows users to define a color for each individual point throughout the whole volume of the model. These points are called voxels – a three-dimensional version of the pixel. This type of printing enables the fabrication of parts with complex material distribution, such as a gradient or a sophisticated pattern. Designing parts for Voxel Printing remains challenging and just a few years ago represented a daunting task. Something that only some of the best special FX artists in the world and some of the best University teams in the world could address. The ability to share knowledge across different teams is of paramount importance to mastering the possibilities that Voxel Printing can offer.
“The critical thing that we set out to achieve was full-color multi-jet printing,” Rob says. “Initially we were working on a Zcorp printer, which was analogous to an inkjet printer, depositing dyes into the gypsum bed. We had used the Objet printers on some of our films, and we knew that they had multi-material capabilities, but they were limited in terms of combining those materials. We did use them in a limited capacity on Coraline where we just had black and white. When we learned that Objet was going to launch a full-color printer, we knew it was going to change everything.”
“It was around the time that we met Nick,” Rob goes on. “There was something about Nick’s talk where he had figured out some things that we were very curious about. This is how these conversations go. It’s partly us showing off, ‘we know how to do that but hey, show us the cool thing that you figured out’. Around that time, I had engaged our VFX team who had a lot of experience authoring Voxels. We came up with a tool inside of another tool called Houdini, which is one of the many that we use to generate visual effects and animate three-dimensional objects, to basically generate these slices. But there still were no tools for designing voxels and no real concepts about how to do it. We were generating a toolset and a framework. The work that Nick was doing already had some conceptually grounded things in the medical imaging field.”
The potential applications of this work can bring benefits to researching motor neuron diseases that have to do with the relationship between these physiological data. For example, Anorexia Nervosa and Parkinson’s are diseases that are a result of multiple factors that we’re just starting to understand.
Progress is going fast. Nick’s lab has access to three hospitals within five minutes: the Veterans Administration Hospital, Children’s Hospital, and Adult Hospital. And over the years, he has built relationships with as many as 35 different surgeons. He has access to the operating room to observe, while surgeons can visit his lab between surgeries. “We have a very intimate relationship and ability to talk to Rob one minute and walk over to the OR, see the problem, ask questions, and have rapid feedback. It’s wild because when I started with this, almost nobody understood its potential,” he tells us.
“It’s the result of some smart people playing around with MRI and understanding it. Up until this point, doctors were starting to look at blood flow and velocity but using computational fluid dynamics. Now we’re able to get it directly from the medical images,” Nick says.
“In architecture—Nick goes on—we can provide a structure for nature to be able to grow in a healthy way. I think maybe the best solution for a lot of our complex areas where people are looking at bioprinting is not actually full bioprinting, but a hybrid between a structured device and one that allows for the human body to grow into and around it. I believe that this interplay between an engineered patient-specific device and the human body may have the potential to fill a lot of gaps and provide a lot of solutions. One of the big areas is looking at patient-specific implants for soft tissues that utilize different parameters, different microstructures, and a lot of the benefits that Voxel Printing can offer.”
Nick had lots of research queued up that could take advantage of these capabilities. At the same time, Rob and his team at LAIKA had conquered everything they could in animation that they could think of. “I interested in a new challenge, and it seemed that Nick he had some very interesting ones ahead,” Rob says.
Recreating physiology
The special effects world regularly works with such data. Nick’s and Rob’s project now is looking at combining all of these different forms of data for surgical applications. One of the first ones they’re focusing on right now is for pediatric epilepsy.
Understanding everything involved before surgical intervention in pediatric epilepsy involves five to eight different forms of data. If a child has intractable focal epilepsy, surgeons need to try to intervene as young as possible because it gives the brain the opportunity to heal. The best solution we have right now is to take out a portion of the brain. Sometimes it’s an entire hemisphere of the brain. Sometimes it’s just a lobe, and sometimes it’s just a small portion of the lobe. “We believe that by giving them more of this data, being able to process and understand it, we can limit the amount of a resection of the brain,” Nick says.
“I got into 3D printing about 10 or so years ago – Rob begins – when we made Coraline, which is the first film that came out of LAIKA. There were two gentlemen who came to the studio to help facilitate that project. They had done a little bit of stop motion with Henry Selick on Life Aquatic and had the bold idea of printing a bunch of faces for the characters using a technique called replacement animation. Typically, those things would be hand-sculpted or sculpted and molded and cast. I was brought in to help them build a workflow and toolsets around that process. Ever since then, we’ve continued to progress in using additive tools, mostly focused on PolyJet printing. But we have many different printers in our facility now. We also use a lot of other computer-aided manufacturing tools to make this age-old process of stop motion animation.”
The ability to see the tractography areas you really don’t want to cut is critical to a surgeon. Up until this point, everybody was looking at this growing field of physiological data on two-dimensional screens. Visualizing tractography and blood flow is very complicated, even in 3D.
In the following interview we will see how Rob Ducey’s experience in “animating miniature beings” proved instrumental in Nick Jacobson’s work with surgeons operating on the human brain, enabling both to continue to expand their, and everyone’s, understanding on how to create the most complex 3D printed parts possible, based on multiple data streams originating from different sources.
Nick set about on this journey of trying to recreate that and became an early adopter of bitmap technology. Objet came out with a research package that they gave to just a couple of universities, and Harvard was one of them. They basically said, ‘Hey, you can bypass all the internal slicers, see what you can do with it.’
“From my point of view—Rob says—technological development is going towards more channels and dimensions of control. The J850 gives you seven different materials from which you can generate pretty much a full gamut of color with just five materials, C, M, Y, K, and W. You can add translucency and transparency for the ability to see deeper into objects. We also want to add in this ability to have various parameters, so you can have something that is tactically flesh-like. But in order to do that at the same time as having full color, you need to add additional channels to control those dimensions.
Research on Voxel Printing capabilities may help address this as well. In fact, Nick says that in a way, this has been the ultimate goal of his work. “I worked with Jennifer Lewis and the LewisLab and I was working at the Wyss Institute for Biologically Inspired Engineering. She was also a member of the lab, Nick tells us. “Bioprinting organs is very far in the future but here’s my sci-fi idea: we have funding for this, and it’s an active area of research to create patient-specific implants that utilize this technology for soft tissue applications. And as an architect, I come at this from a little bit of different perspective.
“He was Mr. Open Source. He brought over his surgeon to MIT and to Harvard and said, ‘There’s all this great technology out there. Is there anything that could help me help my surgeon?’ They engaged my cohort of bitmap printing researchers and made a model of Steven’s brain with the tumor. They worked side by side with the surgeon to design a tool that would help him perform the surgery. It was a great success. Steven had the surgery, was back working on his dissertation in two weeks. It was incredible. He was in the New York Times. He went and met the President as this kid who was using all the tools at his disposal.
“My interest in 3D printing really took off when I went to Harvard University’s Graduate School of Design,” Nick continues. “I focused on computational design and digital fabrication, looking for opportunities on both sides. I got into bitmap printing through a study of structure. When we would do a structural analysis in the computer, it never looked like the buildings that we were building with solid steel and wood beams, it was this more interesting form. There was no way that you could possibly build that but, if you could, it would open up new opportunities. It would also create a better structure.
Voxel Printing gives users access to the printer’s core operations. A PolyJet printer voxelizes a model figures out where the materials need to be, and makes a map. With Voxel Printing users have access to designing those maps and can determine the material configurations throughout the entire volume of the object.
“I thought: ‘What could they possibly be doing?’”, Rob says.
It may be difficult, at first, to envision a collaboration between a special effect artist and a surgical modeling expert but in 3D printing, this is not only possible, but it is also almost natural. One of the main goals of the AMUG conference, the North America-based leading global event for AM users, is to make sure that these professionals from different adoption segments can meet, and exchange experiences, thus enabling the entire AM industry – and, more generally, human capabilities – to advance.
“Without Voxel Printing—Rob explains—you would have to have multiple objects that overlap each other. They would be very, very dense models where the system has to work out dominance and overlap. Working in the visual effects industry, we have access to all these visual effects tools which have so many ways of dealing with volumetric data. We realized that we now had the full capability of authoring volumetric data that we can then feed to these printers and try some really interesting things.”
“There are other areas—Nick continues. For example, tractography focuses on the areas that connect certain elements in your brain that are critical to know for surgery. There’s a series of tracks or neurons that connect the two language processing centers in the brain, the Wernicke’s and the Broca’s areas. If you have a tumor in one of those areas, it’s the scariest thing a neurosurgeon has to do because if you damage the language processing center of the brain, your ability to speak or process speech, it’s shown to be the most debilitating thing that can happen to a human.”
This is what happened when Nicholas Jacobson, Translational Research Faculty at the University of Colorado Anschutz Medical Campus, and Rob Ducey, Technical Supervisor at LAIKA, met at AMUG, learning about each other’s work in the “fringe” field of Voxel Printing (formerly Bitmap printing). Their amazing accomplishments in combining bitmap printing for special FX animation with surgical preparation will be presented as one of the opening keynotes at the upcoming AMUG Conference in Chicago on March 19th.
“The amazing thing about Voxel Printing is that it allows us to represent and model soft tissue,” Nick tells us. “Because soft tissue is made up of different scales of an organization that blend, it’s hard to understand the hard boundaries, for example, in the brain, the kidneys, or the liver. Bitmap printing opens up that whole world of soft tissue application, where things change point by point in space. We’ve gotten really good at that in my lab.”
“We used to think that it was one simple thing gone wrong, and now we’re understanding that it’s the relationship between various forms of physiological data. We believe that by putting this together, we can start to mine this and understand some of these underlying problems.” Nick concludes.
The future of voxelized medicine
“We were starting to get this type of data, and I had come up with some very difficult and bootleg ways of visualizing it and bitmap printing, but they weren’t great,” Nick says. “We now had 4D flow, tractography, fMRI, and the field was growing. It seemed like every day there was a new type of data. How do we print that? That’s where I started to engage Rob.”
Rob Ducey, Technical Supervisor at LAIKA.
How it all started at LAIKA
Stay up to date with everything that is happening in the wonderful world of AM via our LinkedIn community.
As the technology becomes more standardized its use will become more widespread across hospitals around the world. Nick tells us that the FDA is researching 3D models to understand if they can be approved for insurance coverage, which would mark a major change. “We’re gathering all this data during trials and—he reveals—if we can prove that there is a benefit, insurance companies will reimburse them. And we see a lot of companies lining up right now, getting ready for that likelihood. And a lot of companies talking about having a print center in every hospital.
Those who eat, sleep and dream 3D printing know that one of the most fascinating aspects of this new way of manufacturing is that it can span across segments, blending and advancing them together. Lessons learned in aerospace are implemented in the energy segment; designs developed for consumer products may be used in architecture. Sometimes these shared elements can span across very different segments: at the crossroads is where some of the greatest steps forward take place.
Nick met Rob at AMUG. He and Brian McLean, Director of Rapid Prototype at LAIKA, were giving the keynote speech and they were talking about using it in their Academy Award-winning movies. Later that day he also gave a talk on bitmap printing, and they started talking and sharing experiences, exploring how they could use this technology to create something that was really exciting and useful, specifically for medical applications.
“After seeing that, it was really hard for me to think about ever going back into architecture,” Nick says.
