The Camera of the Future
June 18, 2018
A camera reference design from chip maker Socionext. The company sees machine learning playing a growing role in a forthcoming wave of smarter cameras.
At the dawn of digital photography, it was common to hear people refer to digital cameras as nothing more than computers with lenses on them. It was often a term of derision—digital as the interloper in the sacred interplay of light and chemistry. But others had a more optimistic view: As computer processors and the algorithms that ran on them improved, as they invariably do, it would enable cameras to do incredible things.
We now know which interpretation carried the day. In the 20 years or so since the advent of the digital camera, the computer behind the lens has enabled some pretty incredible feats—from automatically finding and focusing on faces (even eyes), to extreme low light sensitivity, to writing huge amounts of data to memory cards as fast as your camera can capture it.
So what comes next? According to the companies tasked with building those camera computers, aka image processors, the future promises even better noise reduction, more efficient video compression for high frame-rate 4K, 8K and livestreaming, advanced image stitching and the emergence of smarter cameras, powered by machine learning. “The digital camera revolution started with the image sensor, but it’s increasingly about computation,” observes Jim Merrick, Director of Marketing at Qualcomm Technology.
Take noise reduction. Today’s cameras are capable of very high ISOs, but combatting noise isn’t simply about achieving astronomical ISO values. It’s also a function of the continued push for ever higher resolution image sensors, says Mitsugu Naito, SVP and Head of Imaging Solution Business Unit at the chip maker Socionext.
Far from ending, Naito sees the megapixel race continuing, putting pressure on processor makers to cope with more noise. “As the pixel size becomes smaller while the physical size of the image sensor remains the same, noise reduction becomes ever more important,” he says. Faster shooting modes also have a tendency to introduce random noise in images, which is also leading chip makers to focus on noise reduction technology, Naito adds.
Sony has been beating back noise and coping with higher-resolution image sensors by reworking the architecture of its sensor and processor, using back-illuminated sensors in a stacked structure with an added front-end processor to improve speed, sensitivity and image quality, a company spokesperson told us.
Improved processors will also enable cameras to migrate to new video codecs, such as the more efficient HEVC/H.265, allowing cameras to record 4K video at 60 fps or even 8K video, Naito says. HEVC will also enable traditional cameras to perform real-time video streaming wirelessly, he adds.
With HEVC, cameras can enjoy a 40 percent decrease in video file size over H.264 but at the same time, higher bit width and better image quality, says Qualcomm’s Merrick. However, HEVC is more computationally intensive to encode than H.264, so it requires processors with plenty of horsepower. “When we do benchmarking, video encoding is the most intense of the benchmarks,” Merrick says.
Only a few traditional cameras, such as Panasonic’s GH5, offer H.265, but it’s increasingly prevalent on smartphones. Indeed, smartphone chips are increasingly pioneering photo technology in areas such as image stitching, depth sensing and working with multiple sensors and lenses, Merrick says. The company worked with (now defunct) Lytro on its Lightfield technology and with Light on the multi-camera L16. “These products are driving a sea change in how we think of camera technology. It’s no longer a single sensor,” he says.
The next wave of camera innovation will focus not just on typical photo parameters like resolution and image quality, but on boosting camera intelligence in general, Naito predicts. “Current cameras already have analytical functions such as face detection and scene detection, but new cameras will require more intelligence and machine learning technology to help drive in the areas of image analytics,” he says. “For example, a software algorithm can be incorporated to identify particular person(s) of interest from a large group of people or have the ability to add special effects to images and at the same time have the ability to transmit images along with the metadata, wirelessly.” The company sees “great potential” in the use of AI in image processing, Naito says. For its part, Sony was coy about using AI in future products but tells us it sees “various possibilities” for incorporating it in the future.