A new vision

Neuroscientist Sheila Nirenberg’s breakthrough neural code is restoring sight in humans and expanding visual capabilities in computing, but some of her academic peers would like to see more transparency in her data.

A new vision
[Photos: Justin J Wee]

On the eighth night of Hanukkah last year, Barry Honig saw light. He could see the shape of the menorah, the flames like fuzzy buds at the top of long sticks.


Honig owns and operates two businesses but hadn’t been able to see a menorah for 20 years. In his younger years, he could discern a brunette from a blonde or see the sun shimmering on a lake. But by his thirties, his ability to see detail had started to fade.

He was born with a retinal disorder called Leber congenital amaurosis (LCA). Inside his eyes, photoreceptors, the cells responsible for translating light, were dying off. It was as though someone had put a dimmer switch on his whole field of vision and dialed it way down.

In the spring of 2020, a friend mentioned a neurologist named Sheila Nirenberg, who was running a clinical trial that might restore vision in people with retinitis pigmentosa (RP), the greater category of disorders that LCA falls within. Honig emailed Nirenberg immediately.


Within minutes, Nirenberg replied: “What is your eye condition? Have you ever seen? Do you have sight? How much can you see?”

“We had a whole email conversation on this Saturday night!” Honig says.

Nirenberg sent him to an ophthalmologist for preliminary tests, then met with him to conduct a baseline evaluation. On September 3, 2020, Honig received an eye injection with a light-sensitive protein and became the second person in Nirenberg’s clinical trial. Three months later, at home, Honig saw the light emanating from the menorah.


Nirenberg’s novel treatment combines gene therapy with a pair of computer-assisted glasses, similar in style to Google Glass. The format itself is not unique—other companies have used proteins and goggles to try and bring back sight. But Nirenberg says her device communicates visual information to the brain using the same language cells that those within the retina use. If the trial proves effective, it could restore vision to the roughly 2 million people worldwide who suffer from retinitis pigmentosa, and millions more if it works for other forms of blindness. Nirenberg’s neural code could also change computer vision as we know it.

“People think of the eye as being a camera and that the retina’s just the film in the back and the brain does all the hard stuff, but that isn’t true,” says Nirenberg. “The eye is actually a little mini image processor.” Through evolution, the eye has figured out which features from the visual world to pull out in order to perform the most basic daily tasks—from recognizing faces and objects to maneuvering around a room—and sends them to the brain. In RP, when photoreceptors die off, visual information can’t get in. Normally, a series of neural circuits between the photoreceptors and what are called ganglion cells translate the visual world into code the latter can understand and send to the brain. Scientists have tried to recreate this chain of communication using genes or electrodes. Through this approach, patients have been able to detect light and shape, but normal vision remains out of reach.

Nirenberg has devoted more than 20 years to research on vision. She’s published dozens of papers in peer-reviewed journals, won more than 20 patents, garnered a MacArthur “Genius Grant,” holds a chaired professorship at Cornell University, and helms two startups to develop applications for her work in humans (Bionic Sight) and computers (Nirenberg Neuroscience). Her backers, all angels, range from wealthy RP cure seekers to a retired Goldman Sachs banker.


But as Nirenberg’s commercial ambitions have advanced, her pace of publishing has slowed. Her last peer-reviewed paper appeared in 2018. Of course, the 700-page clinical trial application she wrote was reviewed and approved by the Food and Drug Administration (FDA) between 2019 and 2020. She also recently presented interim results for her clinical trial at the conference Gene Therapy for Ophthalmic Disorders in September. Still, some academic colleagues wish she’d shine more light on the details of her neural code.

“People in the field, they’ve been skeptical because she hasn’t shown any of the nuts and bolts of how it works,” says Connie Cepko, Bullard Professor of Genetics and Neuroscience at Harvard Medical School and a Howard Hughes Medical Institute Investigator. Cepko also sits on the Scientific Advisory Boards of Permeon Biologics, Advanced Cell Technology and GenSight Biologics, one of Bionic Sight’s competitors. Nirenberg worked in Cepko’s lab as a graduate student.

“If you share it with the world,” counters Nirenberg, “you can never actually bring it to people who need it, because by definition you become unpatentable. If you’re unpatentable, you can’t raise money.”


The standoff over patenting versus publishing has escalated over the past few decades, as private-sector funding for medical research has soared. But the concomitant de-emphasis on peer-review publishing can create a lack of transparency that leaves observers wondering whether they are looking at a scam like Theranos or a genuine breakthrough, like CRISPR.

If Nirenberg’s clinical trials and her computer vision startup prove successful, the questions about her lack of transparency may recede. But the larger question for the $750 billion global biotech industry is how it will fund future Modernas and Gileads and da Vincis without robbing science of its access to data and to studies that drive incremental advances. Open, basic research has delivered CRISPR, mRNA vaccines, and penicillin. As more scientists find fortune in patents, will science itself become all the poorer?

Nirenberg was born in New York City, the daughter of a psychologist and a poet, though she spent much of her young life in suburban Edgemont, a small community about 20 miles north of Times Square. She was the middle child of three girls, with a “super achiever” older sister, something she considers lucky. “Nobody was paying attention to me—except for, in a normal, loving parental way—so I was free to invent things and have ideas,” she says.


As an undergraduate at the State University of New York at Albany, she studied literature, winning a university award for a short story as a freshman. But in her senior year, a class on genetics led her to consider the sciences. After solving a contradictory problem about DNA, she claimed her first research award. Suddenly, the possibility of becoming a scientist felt ripe.

After years of diminishing sight, Barry Honig has regained vision. [Photos: Justin J Wee]
“I really wanted to be a writer, but writing, it’s very internal,” she says. “You’re looking at yourself all the time. And science is external. It doesn’t matter what someone thinks of you. If you’re right, you’re right, you have evidence to prove a case, and I just loved that.”

Nirenberg chose the brain as her area of focus. She worked in a lab as a technician for a couple of years before applying to PhD programs in neuroscience. She landed at Harvard Medical School, where she had to design a project that would add a gene to something, but she didn’t think the assignment made sense. “What do you learn about that gene? All you’ve done is disrupt something in a weird way by adding something,” she says. “The better way to understand something is to take something away.”


She set out to target and delete a class of cells as a way of understanding their specific role in a retinal circuit. Nirenberg ultimately figured out that she could make the cells she wanted to kill express an enzyme, releasing a fluorescent dye that when hit with light, would die. She had to turn the enzyme toxic in a very precise way—it couldn’t spread and kill the wrong kinds of cells. The work took her a few years and many failures to resolve.

“It’s very hard to go into a circuit after it’s developed and just remove one element of it,” says Cepko, recalling her former protégé’s early work. She worried that Nirenberg would burn out working on the problem, and urged her to give it up. Who knew if it would even work? But Nirenberg persisted. “She just hung in there, stuck with it, got it to work.”

After earning her PhD in 1993, Nirenberg moved to another lab at Harvard College for her postdoc. With just a few published papers, she got an assistant professorship at the University of California, Los Angeles, where she continued her work in neuroscience. The cell-deletion tool she made allowed her to study the way cells in the retina interact with each other to process information.


In 2012, Nirenberg was working at Cornell University when she published a paper in the prestigious peer-reviewed journal Proceedings of the National Academy of Sciences. The paper built on an existing approach to prosthetic vision, using a light-sensitive protein found in algae called channelrhodopsin to reignite light receptivity in the retina. The process for injecting the channelrhodopsin gene into the eye is called optogenetics.

In Nirenberg’s model, an optogenetic gene is inserted into the eye, reanimating the retina’s ability to perceive light. A wearable device sends in light signals, coded in language the ganglion cells can comprehend, to the retina; this is then transmitted to the brain. This code, she wrote in her seminal paper, would enable her to make prosthetics capable of bringing impaired vision “into the realm of normal image representation,” far exceeding the performance of existing therapies.

She filed a provisional patent before publishing, and founded two companies, the first for restoring vision and a second to explore the uses of this neural code in robotic vision. At the same time, the MacArthur Foundation granted her a “Genius” award, as a member of its fellowship class of 2013.


The peer review versus patent issue has grown increasingly controversial in academia. Since the 1980s, following the Bayh-Dole Act, universities have been pushing scientists to generate scientific intellectual property that can be patented to generate profits, which can then be put toward new research. The University of California, Berkeley, famously used Bayh-Dole to get government funding for research into cancer drug Yervoy. The money generated from that patented research went on to fund future work, including Nobel laureate Jennifer Doudna’s CRISPR research and patent.

The incentive for profit has changed the open nature of publishing scientific research. Scientists who invent something truly novel are encouraged to patent first and share just enough detail in their published work to satisfy the peer-review process. Bare-bones papers can lead to frustration in the community and, critics say, limit opportunity for validation, creating false hope for people suffering from disease. The U.S. patent office also relies on published papers to validate worthy technology.

Cepko, who’s working on several methods for preempting the death of photoreceptors in people with retinitis pigmentosa, believes Nirenberg’s process is plausible, but would like more detail on what happens after the optogenetic code has been introduced to the eye, as well as the basis of the code itself.


“If she’s really got something, she’s just got to figure out a way to let people see her data,” says Cepko. “That’s just the lifeblood of science, right?”

Nirenberg insists the money is a means to an end. If she really wants to bring this invention to people who need it, she has to pursue the patent and private funding route. She learned this, ironically, while working under Cepko. When Nirenberg invented the cell-deletion tool during her PhD work, she presented her findings at a conference. “I thought everything was about sharing,” she says.

Then Harvard’s tech transfer office, which patents the work of its students and teachers, called her in for a meeting. They were confused, she says, about why she had both published a paper about her work and presented her findings at a conference. Nirenberg says she was bewildered. “I had an invention. I thought, that’s what you do,” she says. “They said, ‘No, you have to file a provisional [patent].'”


Sight Lines:How Bionic Sight’s gene therapy and headset work together to help restore a patient’s vision

Click to expand [Illustration: Kouzou Sakai]
It was an important lesson in the business of science. “The amount of money that you need to actually make a treatment is well beyond what a typical grant is: $250,000 a year to pay for your starving graduate students and yourself,” she says. Whereas, to get Bionic Sight to where it is, she’s had to raise $13 million, predominantly from angel investors. To pay for mass manufacturing of her viral gene and production of her hardware headset will require far more.

Mammoth Bioscience and Scribe Therapeutics, two of three companies founded by CRISPR pioneer Doudna, have raised $118 million and $121 million, respectively, according to PitchBook Data. Doudna’s third company, Caribou Biosciences, went public over the summer, raising $304 million for the venture—hailed as one of the most lucrative deals in the gene-editing space.

Biotech is increasingly becoming a hybrid of software and science. A key part of Nirenberg’s work is code, which can be validated outside the peer-review process. Her second company, Nirenberg Neuroscience, is bringing human sight to computers. “If this code was so useful to our brains for processing information,” she asks, “what if I teamed up with computer vision and could make robots see?”


What is different about Nirenberg’s code is that it doesn’t need huge amounts of training data to understand the visual world the way another current approach, deep learning, does. “I don’t need all the details of you: I don’t need to know where every eyebrow hair is. I don’t need to know how you’re represented in every shadow,” she explains. “I capture the essence of you and that’s what the retina does for the brain.”

Inside a robot, the technology works the same way. It literally does the math, turning the visual world into a readable code. Where deep learning makes robots compare video against a library of snapshots, Nirenberg’s code pulls out important features the way the retina does. It can also see in varied lighting and weather conditions.

In 2016, Nirenberg Neuroscience licensed its vision technology to Ford to put into self-driving cars, but within a year Ford struck a deal with Argo AI for its comprehensive self-driving system. Ford could still integrate Nirenberg’s computer vision into Argo’s technology, but it would be difficult since they were built on two entirely different systems.

Intel signed a collaborative agreement with Nirenberg Neuroscience in April 2020, though the company has been vetting Nirenberg’s technology since 2019. Stacey Shulman, a VP in Intel’s Internet of Things group, says scaling Nirenberg’s code for Intel’s self-driving cars would take too much time, because it would require deep integration with a bevy of other systems. Instead, Intel is piloting the technology as a training tool for handwashing inside major fast-food chain restaurant kitchens. A sensor is angled over the sink, while a tablet fixed at eye level plays an animation of cars driving across the screen until the program deems the handwashing sufficient.

Shulman has also run a pilot with a large retail chain to see if Nirenberg’s code could spot instances of intentional theft (as opposed to opportunistic stealing). Organized retail crime costs retailers an average of $719,548 per $1 billion in sales annually, according to the National Retail Federation. Using deep learning to identify theft is difficult, because it is trained on thousands of images of people shoplifting and runs the risk of creating the wrong associations between a specific demographic and stealing. Shulman says Nirenberg’s code can look at behavior rather than physical traits. “We’re simply looking at how they move,” she says. “What we were able to determine is that based on the speed that somebody picks up an item off the shelf, and whether they look at it or not, we can determine their likelihood of stealing it.” Also, because the networks in Nirenberg’s technology are very shallow, the decision-making is easy to trace and understand, says Shulman.
She gets why Nirenberg doesn’t want to share her code. “Once it’s written, it’s not hard to copy something that’s digital.” Nirenberg has of course patented her technology, but she’s still a tiny company. In total, Nirenberg Neuroscience has raised $2.5 million, though additional funding has come through licensing deals and contracts.

“It’s also a big difference, between a company the size of an Intel and the legal team that we have and the size of a Bionic Sight or Nirenberg Neuroscience,” says Shulman. “Does she have a hundred attorneys on staff? No.”

In terms of where this technology could be used, Shulman says the field is wide open: everything from helping surgeons be sure an operating theater is set up before a surgery to making self-driving cars see. “I don’t know how many digits of billions,” Shulman says, trying to estimate the market potential for Nirenberg’s code, “but it’s in that realm.”

The treatment for the Bionic Sight clinical trial is a two-part process. The first step involves injecting the gene into the eye. Once screened and approved, participants go to OCLI Vision, an ophthalmology practice on Long Island. Clinicians lean patients back in an exam chair and use a topical anesthesia to numb their eyes before the injection. Patients feel only a bit of pressure in the back of the head during the procedure.

The clinical trial was approved to test the gene at three doses. Barry Honig received the lowest dose, last September. The second-strength dose was given to a new cohort of three patients toward the end of April, and a third group of three was administered the most potent dose in June. (The first three phases have included a total of eight participants—finding suitable subjects was complicated by the pandemic; Nirenberg hopes a fourth phase will bring the total number of trial subjects to 20.)

The second part of the trial involves a desktop machine that converts images into neural code and sends it to the retina. Participants peer into an eyepiece at a screen and hit a series of buttons to indicate whether a bar is moving left or right, or to identify an object—such as whether an image is an apple or a banana.

About 15 weeks after receiving his injection, Honig nearly jumped back from the eyepiece when the lights came on. He says it was bright, like the flash on a camera. In actuality, the light was 40 times less intense than the light he was presented with during his baseline test. “I was literally tapping the buttons like I was playing a video game,” he tells me. “In other words, there was no doubt that I was seeing the thing.”

What this meant was that the gene had taken—an important first step in potentially advancing Honig’s vision. It also meant that the device was able to communicate the visual information in front of him to his brain. For now, the device is the size of a tabletop, but eventually it will be embedded into a pair of glasses.

Nirenberg is now amending her clinical trial to add a fourth dose of the gene to her study. With each dose, the retina becomes more light sensitive. In studies on mice, this fourth dose had a significantly higher impact than dose three.

At this level, she believes her device can achieve near-normal vision. She’s hoping to get the FDA to sign off on the fourth dose soon so she can complete all dose levels by the end of the year. Considering the safety profile so far, she’s optimistic.

Nirenberg is currently seeking venture capital on the belief that Bionic Sight will make it through the clinical trial and gain FDA approval. With that certification in hand, she’ll have to focus on scaling the business to manufacture the optogenetic gene and the accompanying headset. “I’m getting all those pieces in place to build up to that bigger structure,” she says.

She finds herself in a position familiar to other academic researchers whose breakthroughs show tremendous medical—and commercial—potential, and she’s unsure whether she’ll sell both businesses or try to build them out herself. Either way, she’s keen to get back into the lab.

“It’s not the business part of running a biotech company that is the excitement,” she says. “I just want to be the scientific director in there doing stuff.” But she’s also aware of the responsibility of creating a piece of technology that could make robots see. Jotted down in her calendar at a future date at 10 a.m. is the phrase “30 pieces of silver,” a reference to the prize Judas received from the Romans for betraying Jesus. It’s a reminder to ponder the ethical balance between money and her duty to protect her code from harmful uses, and to ensure that her prosthetic retina doesn’t get acquired and killed. “That’s part of why I want to keep running my own company.”


About the author

Ruth Reader is a writer for Fast Company. She covers the intersection of health and technology.