How holograms, deepfakes, and AR are raising the dead

In recent years, many celebrities have been raised from the dead in the form of ghostly “holograms,” including Whitney Houston, Marilyn Monroe, Ronald Reagan, Vince Lombardi, Michael Jackson, Roy Orbison, Frank Zappa, Elvis Presley, Amy Winehouse, Glenn Gould, Maria Callas, Buddy Holly, and Ronnie James Dio.

Everybody calls these artifacts “holograms,” but that’s a misnomer. By definition, a hologram is a 3D projection that can be viewed at different angles. These recreations are 2D projections that require the audience to be sitting in the right location.

Creators of these celebrity “hologram” performances hire an actor or impersonator who looks and moves like the subject of the hologram. Once the footage is captured, the face is replaced by a deepfake video that looks like the subject, and visual effects specialists touch up the combined video to remove some of the glitches and imperfections generated by the current state of the art in deepfake technology.

This is the effect that Disney has long used for Disneyland’s Haunted Mansion attraction. Ghosts appear to be dancing in a room. The room is visible through the glass. The “ghosts” are mechanized and dancing below the visitors, but reflected in the angled glass that sits between the guests and the room.

The state of the art in hologram projection uses a product called Holonet from a company called Kaleida. Holonet is an invisible gauze that reflects light more brightly than glass. It’s the technology used in the Kardashian hologram, as well as the Tupac Shakur event at Coachella in 2012. Gauze ghosts haunted seances in the 1920s; now they continue to haunt concert halls in the 2020s.

In the case of Shatner’s “ghost,” StoryFile CEO Heather Smith told me that part of the company’s core mission is to produce faithful, unedited, unmodified video of their subjects—there is no deepfake technology involved, no fake anything. The “AI” used is for retrieval of the right video segments based on a user’s question to the subject in the video. While StoryFile is working hard to make its technology available to everyone using low-cost, low-tech media, the high-end version used in the Shatner recording provides future creators with all they need to make actual 3D holograms.

Another difference between Shatner’s hologram and other recreated celebs is that he consented, whereas most other subjects did not. There are ethical implications to profiting without permission from the likeness of the dead, something that authors like Elaine Kasket, author of All the Ghosts in the Machine, have explored deeply.

So the celebrity “holograms” you’ve heard about aren’t holograms. And they’re also not ghosts.

[Source video: MART PRODUCTION/Pexels]

What is a ghost, anyway?

The ancient idea of a ghost is that our physical bodies are meat puppets animated by an invisible, nonphysical “mind” or “soul” or “spirit.” When our bodies die, the idea says, our souls do, or can, persist.

I would argue that a “ghost” is not a soul in the abstract, but the perception or the experience of the presence of one of these “spirits.” It’s not a “ghost” unless someone experiences or perceives it.

That’s why seances were so convincing. Show a grieving human a glowing wad of gauze, and voilà, grandma has risen!

Ghost sightings require a trigger. The seance hoaxes of the 1920s often involved photography. Fraudulent medium Eva Carrière used fake “ectoplasm” made out of chewed-up paper, and photographs that people believed were ghosts. In one famous photo, Carrière is exposed materializing the “ghost” of King Ferdinand of Bulgaria, which is obviously a cardboard cutout.

We’re on the cusp of a new world of technology that will be capable of inducing not only triggers, but ultra high-resolution, interactive, 3D versions of the dead that could bring back a new age of Spiritualism.

The technology behind the perfect ghost

Apple is reportedly working on two augmented reality products. The first is likely to be a set of goggles that will mostly do VR. The real world and the virtual world will be blended together, then displayed on two very high-resolution screens built into the goggles.

In the second product, the real world will be seen through clear glass, and virtual objects will appear like holograms in the real world. These will be designed for all-day, everyday wear.

Dozens of major companies may eventually make such glasses, but Apple will mainstream the technology. It may only be a matter of time before normal reality is always augmented for many people.

Unlike the Tupac and Kardashian projections, consumer AR glasses will enable simulated 3D holograms—different people viewing the same virtual object will see it from their own perspective.

Deepfake audio is already nearly perfect. And Deepfake video is getting better every year.

AI in general—and generative adversarial network (GAN) technology, the machine learning frameworks behind all deepfakes in particular—combined with AR will result in the ability to reproduce in 3D and in the real world the face, body language, mannerisms, voice, and patterns of speech of any person that can persist after death. They’ll permit the creation of a dead ringer, if you will, of a real person.

These digital simulations will be made interactive, so you can hold conversations with them. The opinions, beliefs, and knowledge of actual humans may also be uploaded into the mix, augmented with real-world knowledge.

Here comes the digital ghost industrial complex

I believe that AI-constructed digital humans experienced through AR glasses will become the primary way the deceased are memorialized and remembered. Shrines will be constructed in location-based mixed reality. At minimum, they’ll feature hologram-like videos of the deceased, standing there waving as ghostly apparitions and interactively conversing with surviving family members.

In fact, this basic concept is already here. A Japanese tombstone-engraving company called Ryoshin Sekizai offers an AR app called Suma Tomb that creates an AR “hologram” of loved ones that can be placed anywhere—a gravesite, in the home, or at the place of death. The ghosts are visible only through a smartphone at present. But as with all smartphone AR, it’s just a precursor to the AR glasses revolution. As with Kim Kardashian’s dad, the AR ghosts tell loved ones that “we’re always watching over you.”

A South Florida company called Artistry in Motion Holographics offers the recording and performance of “holograms” at funerals, enabling people to basically eulogize themselves postmortem.

The latest ghost technology that animates dead comes from the online genealogy company MyHeritage. Its Deep Nostalgia service uses AI to animate uploaded photos, so that grandma is looking around, blinking, and looking somewhat alive. It’s flawed and creepy, but also moving.

Testing new photo animation features at @MyHeritage. I was 9 when my dad was killed. Seeing him move, even artificially, gives me chills. “Kids today” don’t understand that back in the day moving images were less common, and most folks my age have precious few. pic.twitter.com/0uTnrH3yRW

— Kostelník Juarez (@DellaSeattle) May 2, 2021

In 2015, a Belarusian tech entrepreneur named Roman Mazurenko was killed in a Moscow car crash. And so his best friend, Eugenia Kuyda, recreated Mazurenko as an AI chatbot. She also created a chatbot product called Replika, which asks users questions in order to learn how to recreate the speech patterns of that users to simulate them in chat.

These and other similar projects and products suggest a potential market for interactive digital versions of the people we have lost. Ghosts, in other words.

A return to the spiritualism of the 1920s

It’s reasonable to expect the rise of services that offer Siri-like personal assistants, but represented by digital versions of deceased loved ones. These digital ghosts will serve the same purpose as the seance of the 1920s: to reconnect with the dearly departed, and also get answers and guidance from more evolved beings.

There is also no doubt that AI and AR will be used not only to simulate the dead, but also for hoaxes to convince the gullible that ghosts are real and are speaking to us. Fraudsters and mediums will offer to channel lost loved ones, and will appear to do so through deepfake photography, audio, and video.

Nobody’s talking about it. But I think we’re on the verge of a new era in technology-enabled Spiritualism. Technology will not only raise the dead, but also countless ethical questions about keeping the deceased around as digital ghosts to haunt the living.