Video & Filmmaking

Frames Per Second: How Photographers are Using Apps to Bring Augmented Reality to Book Projects

January 22, 2018

By Holly Stuart Hughes

Looking for New ways to provide context for their images and attract new audiences, several photographers in the past year have produced photo books that embed links to video. The technology, which publishers call AR (augmented reality) functionality, allows readers who have downloaded an app to their smartphones or tablets to stream film clips and audio as they hold their devices over a printed photo.

For example, the 2016 reissue of Susan Meiselas’s 1981 book Nicaragua allows readers to watch clips from two of the photographer’s documentary films, and to learn what has happened to her subjects and photos since the Sandinista revolution. Photographer Lucas Blalock’s 2016 book Making Memeries uses AR technology to display animations he made using the objects he had photographed.

Courtesy of Self Publish, Be Happy


In 2017, photojournalist and filmmaker Nina Berman published An Autobiography of Miss Wish, in collaboration with her long-time subject and friend, Kimberly Stevens. The book incorporates Berman’s photos and Stevens’s drawings, diary entries and medical records, portraying her struggle to overcome sexual abuse. It also uses AR to link to video interviews with Stevens that Berman made and posted to Vimeo. “I include videos because it adds another dimension to the work, to her character, to our relationship and shows some lighter moments,” Berman explains.

To allow readers to view the videos, Berman licensed an AR app from Augment. Like Fetch, Layar, Aurasma and other AR apps, Augment is marketed to retailers who want to give potential customers access to more information about a product they find in a catalogue. It’s also pitched to teachers as a way to let students access more information about a book, an exhibition or a guided tour using their iPads.

Of the many apps an author can choose, all work in roughly the same way. First, authors photograph an image on the printed page that will trigger the streaming video. They upload the image to the app. Then they upload a link to the video they want readers to see. To show readers where to look for trigger points, the linked images are marked with a symbol. In Susan Meiselas’s Nicaragua, for example, a small symbol of an eye appears on the images that would launch videos. When readers hold their mobile devices over a linked photo, the app recognizes the image, and begins streaming the related video.

The technology isn’t new. In 2012, photographer Rick Smolan, founder of the publishing and multimedia production company Against All Odds, and his partner, Jennifer Erwitt, used AR to connect videos to the book The Human Face of Data. A compilation of images and text by several contributors, The Human Face of Data documented how data collected by GPS satellites and digital devices has been used in genetic research, the energy industry, health care, crime analysis and more. To enable video viewing, Smolan contacted Aurasma, a small digital company that was owned by HP. Aurasma became one of the book’s sponsors and in return, programmed an app, available under the name “The Against All Odds app.” “They created a customized version that was branded for us,” Smolan says. At the time, he says, “It worked ok, it was a little clumsy.”

Layout © Against All Odds

A spread from The Good Fight includes trigger points in the corners of two photos, alerting readers to additional AR content. Layout © Against All Odds

He contracted with Aurasma again in 2015, when he produced Inside Tracks, a book about his 1978 National Geographic assignment to follow a woman who had resolved to trek across Australia. Using a mobile device and the Aurasma-powered app, viewers could watch clips from Tracks, a feature film based on Smolan’s National Geographic story.

The newest book from Against All Odds features its most ambitious use of AR to date. The Good Fight: America’s Ongoing Struggle for Justice includes 180 photos showing the struggle for equal rights for women, the disabled, the LGBTQ community, Native Americans, African Americans and other minorities in the U.S. The book also includes links to 60 videos. The production costs of the book and a customized app were underwritten by the Anti-Defamation League, the 100-year-old civil rights group known for its fight against anti-Semitism and other forms of discrimination.

In planning the AR functionality of The Good Fight, Smolan wanted a new, more streamlined app. In the three years between the publication of the Big Data book and Tracks, Aurasma had changed its interface to appeal to teachers. “You had to subscribe to a channel and create an account. It was invasive and slowed you down.” Smolan thought it was too cumbersome for a public used to watching YouTube videos with a single click.

While doing photo research for The Good Fight, Smolan visited the Newseum in Washington, DC., and saw an app that allowed visitors to watch interviews with past Pulitzer winners. He liked the clean interface, so he contacted the developers: Chalk + Chisel, a Baltimore-based company that creates interactive products. In a meeting with Ben Slavin, Chalk + Chisel’s CEO, Smolan explained that he wanted the app to be as simple as possible: “I said if it takes 15 steps, no one’s going to use it. It should be: Just hit one button and it plays.”

In the book, more than 60 photos are printed with a small red circle and an arrow in one corner. To encourage readers to try the free app, there are notes on the book’s title page and back cover explaining that the app is available through Apple’s app store, Google Play and a dedicated website,

According to Smolan, The Good Fight app demonstrates how much image recognition technology has improved since he first used AR. In 2012, the app he used worked only if the page was held flat and the phone or tablet was at a certain angle. Now, he says, The Good Fight app reads an image even in dim light. The book was printed with spot varnish on the photos, so the developers adjusted the software to account for reflections that could obscure image details. The app also allows users to bookmark favorite videos to view without the book.

Some of the videos, such as a clip of Martin Luther King delivering his “I Have a Dream Speech,” are iconic. Others are less known. A photo of an African-American teenager sitting on a curb in Baltimore in front of a riot police launches a conversation, recorded for NPR’s “Storycorps” series. In it, an African-American father says that every time his son leaves the house, he fears what might happen if the boy is stopped by police. The section on Native Americans features an essay and a photo by photographer Aaron Huey, who lived and worked on the Pine Ridge reservation for a year. The photo launches a video of Huey’s talk at a TED conference, explaining his campaign to get the U.S. to honor the treaties it has broken with Indian nations. Says Smolan, “There’s one picture of Aaron’s in the book, but when you see the video, you see there are 50 photos in the talk.”

Smolan and his team made an initial selection of 1,000 videos, found through talking to archivists, documentary filmmakers, photographers and historians. The book’s original budget called for them to narrow their selection down to just 20 videos, “then we got carried away,” Smolan says. As they kept adding more videos, the cost of programming the app increased. “The app developers have to build the links, and then test the app,” he explains. ADL approved the cost overruns and, with their funding, have made the book available for less than $30 a copy.

Smolan believes AR can increase interest in a book. Radio and print journalists who cover technology have contacted him for interviews, for example. “It’s a great gimmick because people who have played with the app have fallen into the world of the book.” He notes, “I have two teenagers, and they’re glued to their phones. Anything they can do with a phone makes a book more interesting.”

Some of the videos tell the stories of image makers who documented civil rights advancements and setbacks. Gordon Parks’s 1956 photo of an African-American mother and daughter standing under a “Colored Entrance” sign in Mobile, Alabama launches a video of Parks describing the risks he faced when he traveled to the Jim Crow South to document segregation. Says Smolan, “The picture itself is powerful, but the video tells so much.”

Notable Photo Books 2017: Part II

Doing the Right Thing: Nina Berman on Ethical Choices

VR Doc About Jet Recycling Shows Medium’s Possibilities

How the UN is Using VR to Show Life after Disaster

Creating a Film and VR Video About One Man’s Experience Going Blind