Video & Filmmaking

Innovations in Visual Storytelling: Moving Beyond the Linear Narrative

July 25, 2016


Eline Jongsma and Kel O’Neill’s “The Ark” premiered at Tribeca Film Festival’s Storyscapes. Before she began working with VR, Jongsma says, “Video and photography always felt constricted.”

Photographers and filmmakers are adapting the technologies of interactive web design, video gaming and virtual reality to expand their storytelling. In an effort to engage new audiences, or draw viewers deeper into stories, they are moving beyond traditional linear narratives. Using a variety of new tools for capturing and displaying still and moving images and sound, these experimental storytellers give their viewers a new, more interactive way to move inside a story. PDN subscribers can learn more about the creation and design of these projects, as well as other experiments in immersive storytelling, this month on PDNOnline.

Frames Per Second: Jongsma + O’Neill Finding Success with Immersive Storytelling 

Through collaborations with programmers and other experts, award-winning filmmakers Eline Jongsma and Kel O’Neill have produced projects in which multiple stories coexist and give viewers a choice of which narratives they follow. The duo have also expanded the audience for their work by creating innovative, web-based presentations. They conceived “The Ark” to be displayed on virtual reality platforms such as Google Cardboard and Samsung Gear VR. It will give viewers the experience of following the parallel stories of scientists in San Diego and park rangers in Kenya who are trying to save the world’s last northern white rhinoceroses. They have continually evolved their documentary storytelling methods since Jongsma, a photographer, and O’Neill, a former copywriter, began combining their talents. “I have always flirted with video, but I never knew how to merge the power of still photography with the documentary form until I started working with Kel,” Jongsma says. “Video and photography always felt constricted, because I wanted to bend the frame somehow. Now we’re doing a live-action VR project and it’s blowing my mind.”

The Interactive Projects of the National Film Board of Canada

National Film Board of Canada (NFB) dedicates 25 percent of its yearly budget allocation to interactive productions. Though it’s government-funded, it has green-lighted productions that investigate government action. In the interactive “docu-game” Fort McMoney, director David Dufresne uses first-person video to take a micro view of Fort McMurray, a boomtown that has sprung up near Alberta’s massive tar sands oil reserve. Players explore the town and hear interviews with residents and business owners about how the local industry affects the town, the country and the rest of the world. The filmmakers’ goal is to stoke educated discourse without pushing easy answers. Dufresne chose the game format in hopes of combatting “green fatigue”: public apathy towards environmental issues brought on by media saturation. NFB’s Hugues Sweeney, the project’s executive producer, notes, “The proposal of Fort McMoney is really about the audience taking control of the city, and making decisions about the city as they go through it.”

Are Video Games the Future of Storytelling?

A few photojournalists have begun using video games as vehicles for storytelling in hopes of engaging new audiences. Butch & Sundance, a media collective based in Amsterdam, launched the On the Ground Reporter series of games told from the viewpoint of a journalist covering a conflict area. “We saw so many ways to tell a complex story better that we grew a bit frustrated because as journalists we were forced to simplify complex and rich stories into 700 words,” explains co-founder Ludo Hekman. “We then only reached the elite who reads magazines and newspapers—that was unfulfilling, too.” The games exist in a purely photographic world: the images and videos shot on location are used to show the people and places the player interacts with. Photographer Marcus Bleasdale also turned to video gaming as a way to show young people how the mining of conflict minerals impacts the Democratic Republic of Congo (DRC), an issue he’s been covering since 2001. “I thought that was kind of an interesting concept: To educate youth, who would play the game, that the very tablets and smartphones that they’re playing on have products that come from DRC inside them,” Bleasdale explains, “and that creates conflict, and that creates child soldiers, and that creates sexual violence, and so on and so forth.”

Karim Ben Khelifa’s “The Enemy” positions a viewer between soldiers from opposite sides of a conflict, who explain what they are fighting for. Khelifa hopes the virtual reality experience will humanize combatants. Image © Karim Ben Khelifa

Former Conflict Photographer Uses Virtual Reality to Humanize Conflict

“The Enemy,” a virtual reality project by former conflict photojournalist Karim Ben Khelifa, aims to reach the next generation of soldiers in seven conflicts around the world and offer them an opportunity to meet their perceived enemies face to face. Ben Khelifa hopes the work will give current and future soldiers a glimpse of the humanity they share with those they fight. Donning a VR headset, viewers of “The Enemy” stand in a simulated room between two fighters, one from each side of a conflict. The VR technology renders the environment and the soldiers in photorealistic 3D, and allows viewers to walk around the space and interact with the soldiers. It’s a concept Ben Khelifa first explored using still portraits he had made of combatants. In 2013, when he first began a fellowship at The M.I.T. Open Documentary Lab, he was invited to try the Oculus Rift VR headset. He realized virtual reality offered a way to bring his subjects to life. “What if the guys that I photographed are actually in the room [with the viewer]? What if they move? What if they breathe?” he wondered. To create the prototype for “The Enemy,” Ben Khelifa interviewed soldiers in Gaza and Tel Aviv using a portable setup that allows a crew of four to transport all of the necessary equipment: Four cameras, simple lights, white seamless, and a “hacked” Xbox Kinect—a motion-sensing device—to capture 3D-images of the soldiers’ movements. Emissive, a French production company, then used all of that data to reconstruct and implement the content in a 3D environment.

Hackathons: Asim Rafiqui on the Value of Multidisciplinary Experiments

Hackathons, originally events for computer programmers to collaborate on software projects, now also serve as laboratories for journalists and photographers who want to experiment. Asim Rafiqui, a Pakistan-based photojournalist who has contributed to TIME, Stern, The Wall Street Journal and other publications, participated in the Bit by Bit hackathon, produced collaboratively between the Columbia University Graduate School of Journalism and The Brown Institute for Media Innovation in February 2014, where he collaborated with game developer Kati London. “Today we are still really only producing what are no more than traditional documentary films, though we call them ‘multimedia,’ and assuming that this is enough,” he writes. “What have not been explored are new story structures, narrative frameworks, image-to-text relationships, greater use of content and images working together, and so much else. That is, what has not really been looked at is how the entire modus operandi of a photojournalist can be redeveloped, and the ways in which entire issues can be covered.” The goals, he says, should be to engage audiences in issues “in deep ways,” explore the political, economic and social forces behind the story, and “create space for a reader/viewer to choose her mode of action.”

Picture Story: Love Radio Rwanda

To explore continuing ethnic tensions in Rwanda more than 20 years after its genocide, Dutch journalists—writer Eefje Blankevoort and photographer Anoek Steketee—created an ambitious web-based multimedia project that mixes fictional drama with documentary images. Called Love Radio, the project was inspired by a popular Rwandan radio soap opera called Musekeweya, that tells the story of conflict and reconciliation between two rival villages. The journalists had planned to use the radio story as a model, “Then we thought, if we’re going to do audio, we should do video, too. It became bigger and bigger, and we decided to do a web documentary,” Steketee says. Love Radio, a web project, includes an “On Air” channel that recreates an abridged version of Musekeweya. For a second “Off Air” channel, Love Radio’s creators explore contemporary life in Rwanda to assess the country’s actual progress toward reconciliation. Realizing they needed interactive design expertise, they teamed up with Sara Kolster, a designer and producer specializing in digital storytelling, and with Kummer & Herrman, an interactive design agency. Together they created the web project, plus a pared-down version for smartphone users, as well as an interactive exhibition.

Beyond the Single Video Screen: Richard Barnes’s Multi-Channel Video Exhibition

Fine-art photographer Richard Barnes’ first multi-channel video installation considers the journeys that undocumented migrants make and highlights the detritus they leave in the desert as they approach the United States border. In 2012, Barnes accompanied anthropologist Jason De Leon and his team to the border in Arizona and made still photographs and did interviews. Working with a video editor, Shared Patel, Barnes produced a three-channel video piece that is projected onto the floor and gives the viewer the sensation of walking through the field of debris found along the border; and a two-channel piece that shows two views of a vehicle driving along the border. The right frame looks at the fence, while the left looks out the front window of the vehicle as it travels along the border in the pouring rain. The exhibition, which Barnes and Krugliak have installed in different art spaces on four different occasions, also includes still photographs, lightboxes with portraits of migrants, and a full wall of backpacks discarded by migrants.

Beyond the Single Video Screen: Mirada’s Immersive Storytelling for IBM

Andrew Merkin is Head of Special Projects & Transmedia at Mirada, a Los Angeles-based production studio that IBM recently hired to create a “Client Interaction Center” for its Watson cognitive computing system. The multiscreen experience gives each client a custom story that details how Watson would help them in their business. “It’s designed to be a seamless experience,” Merkin says. The idea was for Mirada to set up the system so the IBM creative team could then adapt it to create experiences for each individual client. “It was a very intense sprint to get these initial stories up,” Merkin recalls. Limited creative assets existed, so Mirada had to shoot footage, film interviews and create the majority of the content from scratch. “The key to storytelling for [an] immersive experience is that you’re no longer acting as a director being able to frame something,” Merkin says. “So much of it is thinking about the audience, thinking: ‘Where are they looking? Only 5 percent of what I’m doing right now is the focal point of what the audience is looking at, so if I make them turn their head, why am I making them turn their head? If I set off a sound in a different part of the room, I’m going to draw the attention of the audience to that sound. Is that something that I want to have happen?’ It becomes solely focused on the audience or the user themselves.”

Jurying the World Press Photo Multimedia Contest

“Bells and whistles can get in the way of things,” says Keith W. Jenkins, past chair of the World Press Photo Multimedia contest. As HTML 5 was influencing new forms of online storytelling, and ambitious multimedia projects dominated the winning entries, Jenkins said, “Fancy techniques and complicated navigation can be distractions. Most of the projects we saw really didn’t go down that road. Most presented their stories pretty straight.”

See the story in the digital edition (for PDN subscribers; login required). 

CreativeLive Video Tutorial: A discussion on the work of photojournalists and the importance of shooting with integrity and honesty