Preserving History in 3D

Computer Scientist Andrew Jones and I collaborated on his TEDxUCLA talk about preserving Holocaust Survivor's stories in interactive 3D for our grandchildren (and beyond) to converse with. We won a Cicero award in Technology category.

Imagine if you could meet anyone in the world — anyone. Who would you choose? What questions would you ask? And would you record the meeting so that years from now... you could relive it?

As a computer graphics researcher at the USC Institute for Creative Technologies, my main focus has been to make faces, in both video games and blockbuster films, appear less computer generated and more real.

Back in 2010, my team and I were invited to a conference to showcase a project we’d invented — essentially, a floating head in a box.

Now, this display was cool for multiple reasons —

The first was the depth of the face could be seen without 3D glasses. Secondly, this was live teleconferencing where the head could look you directly in the eye, or she could turn away to look at the person next to you... just like in face to face conversations. In fact, it’s these spatial cues that made it feel like my 3D colleague was actually in the room — a feature today’s video chat still can’t deliver.

At first glance, most visitor’s reaction was, “Oh wow, holograms are finally here!”

While others leaned in and asked her to repeat, “Help me Obi Wan Kenobi — you’re my only hope.”

Just as I was getting tired of hearing the same exchange over and over again… a woman from the USC Shoah Foundation grabbed my hand, looked me in the eye, and said “ .. would it be possible to record a 3D conversation... but with a Holocaust survivor? ”

And I thought, I don’t know … would it be?

Andrew Jones, Floating Head in a Box

All we had was this floating head that could only talk in real-time. As a result, the most we’d ever recorded was 60 seconds. That is not much of a conversation. In order to have impact, we would need to display not just a face, but the entire body - head to toe.

And if that wasn’t enough, there was also a tremendous urgency to get this right because the remaining Holocaust survivors are well into in their 80s and 90s — within the next decade many of these firsthand accounts will be lost, forever.

The whole thing felt daunting...

I mean, I was a technology guy — what did I know about the Holocaust? At the same time, how could I pass up the opportunity to have a conversation with a survivor, to preserve a part of history? I knew, in my heart, there had to be a way to turn this 3D vision into a reality.

As we began, I realized we had two major challenges —

The first was how to record a survivor’s story in 3D. While we already had a recording stage — where we turned actors into digital doubles for visual effects — we would need to adapt the stage to be able to record the longer unscripted experiences of real people.

Before filming, I met with our first survivor, 82 year old Pinchas Gutter. As I talked with this warm and friendly man, it was hard for me to comprehend — that between the tender ages of 11 and 16, he had not only survived the Warsaw ghetto...but six concentration camps.

Once we started, Pinchas was incredibly patient as we experimented with different lighting and camera angles. Our dream was that one day... children could gather around 3D Pinchas and see him from nearly any angle to listen to his stories. So we surrounded him with 50 video cameras. By selecting different cameras, we could be compatible with a wide range of displays: conventional 2D televisions, 3D-glasses, or whatever new displays people might be using 20 years from now.

The second challenge had to do with interactivity and the fact that our primary audience would be young kids. While the Shoah Foundation has a library of over 50,000 Holocaust testimonies, most are similar to lectures — which few children hand the attention span to watch.

However, if a child is lucky enough to interact with an actual survivor — and ask questions face-to-face — they feel the survivor’s pain, regret (feel it) and even joy (open, surprised). And it’s these shared emotions that make the lessons stick ... long after the conversation is over.

Which made me wonder, would it be possible to use 3D video to simulate a conversation? Unlike lectures, conversations are typically made of shorter responses, providing gaps for listeners to ask questions.

In order to make this work, we would need to interview Pinchas for many hours — breaking his entire life into these conversational segments.

We started with the most popular questions Pinchas receives when he speaks in public. However, there were also stories we didn’t expect; like when one crew member apologized to Pinchas for the terrible things that had happened to him...and Pinchas responded with this Yiddish joke —

----

When you ask somebody what is the most terrible thing that can happen to you .. and your answer is terrible, nothing is terrible. The only thing is terrible, is when you swallow an umbrella, the umbrella opens inside and you want to take it out. That is terrible.

Questions like these show Pinchas’s playful side.

But there were big philosophical questions too, like — do you blame God for what happened?

Do I blame God, this is a questions that is asked of me many many time, and my answer is that I blame the people, the persecutors, the ones who did all these vile things, the SS, the gastapo, the whole ideology of Naziism That is what I blame for the Holocaust.

Because each answer was spontaneous and different, we couldn’t rely on second takes — I had to make sure we recorded it right the first time, everytime.

What I hadn’t expected was how these heart-wrenching stories would affect me. Like the time, Pinchas told us about his 11 year-old twin sister who was murdered the first day in the concentration camp. What pains him most is...he’s unable to remember her face.

When I heard this story, I was on the verge of losing it. I knew there was audio, which meant I had to keep it together so that no one would hear me sniffling in the background.

But it wasn’t all sad, there were bright moments too — like when we invited children to interview Pinchas. Pinchas would open up, Their questions were so innocent:

Do you have any pets? Or, Have you ever met Hitler?

In total, we asked about 2000 questions over seven days.

At the after party, I made a promise to Pinchas to share his stories, so others could get to know Pinchas and feel his strength and wisdom. But was I crazy? We had recorded so much footage with so many cameras — it could take years to sort through it all.

The key is that one good story can actually answer many different questions. The ICT Natural Language team gathered over 20,000 alternate questions and linked them to different stories in the database. Then we trained a computer to parse visitor’s questions, then play back the closest video response.

Then — five years after this project started — we traveled to the Illinois Holocaust Museum and Education Center, to introduce the new digital Pinchas to the public. While I was still working on 3D displays, it wasn’t ready for prime-time. So we used an 80 inch flat-screen television to show Pinchas just as large as he is in real life.

One day, a group of 15 Polish Holocaust survivor visited the museum. This was the moment of truth, would they think we had justice to their stories? one of the survivors asked “Pinchas, what is your favorite song?” —

As each survivor began to recognize the Polish lullaby.../ they began to sing along - sharing memories of a simpler time before the war.

It also turned out, that one of the survivors had been in the same Polish concentration camp as Pinchas - the only difference was he couldn’t remember a thing. While digital Pinchas could talk about life in the camps, he could never go as deep as the survivor desired — which proved to be the system’s biggest challenge,

The next step is to have a 3D Pinchas in a museum near you. We just built a one-of-a-kind 3D display. Behind Pinchas, you will notice a horizontal band of lights - these are 200 small projectors. As you walk around the display, each projector shows a slightly different view.

Many visitors behave as if Pinchas is really present, apologizing for his suffering or if they interrupt. In fact, one survivor even walked up to touch the screen, and then peered behind it, trying to see where the real Pinchas was hiding.

Since our time with Pinchas, the Shoah Foundation 3D library continues to grow — in fact, we just finished recording 11 more survivor conversations, helping preserve their stories for the future.

Though this journey. I have learned that story and emotions are what really matter. And for me, some of the most emotional moments came when survivors (like Pinchas) were asked —

What message do you have for your family and the future?

“the message i leave with them is that the most important things that they should do is tolerate be human towards each other make sure that they accept any other uh any other person who is of a different culture different religion maybe different color of skin and that they should all live like brothers”

I’m inspired by these words and I can’t help but wonder what wisdom my own father would want to pass on.

Because one day in the not so distant future, this technology will be available to all of us. Our memories will stretch across the generations, giving our great great grandchildren the opportunity to experience our thoughts, feelings, and dreams firsthand.

So knowing one day it will be your turn, I ask you this ... what stories will you choose to tell?