Canon’s Latest AI Image Processor Resolves the Majority of Common Digital Photo Problems

Canon has unveiled an image processing technology that utilizes deep learning to improve image quality in three key areas: noise, color, and lens imperfections. The technology has been developed through the use of a large database of images accumulated through the development of Canon’s cameras and lenses, covering a wide range of subjects.

Using a deep learning algorithm trained on this extensive dataset, Canon has created a software that can correct issues such as noise, moiré, and blurring caused by lens aberrations. The software has been designed to not only understand how camera settings affect image quality, but also how to correct them, using Canon’s unique know-how.

Canon acknowledges that while cameras can capture wonderful moments, there are often unavoidable problems with image quality, such as grainy or blurry photos. To tackle this issue, Canon has directly tackled the development of deep learning image processing technology. The result is a software that can improve image quality and enhance the overall photography experience.

Deep Learning with Three Main Targets

Canon’s deep-learning image processing technology — described as a neural network, which is a type of artificial intelligence (AI) — covers three areas: noise reduction, color interpolation, and aberration diffraction correction.

Noise reduction or removal technology isn’t new, but Canon says that when noise is removed, typically details are lost in exchange and — in most cases — the image quality deteriorates. Using deep learning, Canon says that it was able to obtain clear, high-definition results with little deterioration after removing noise.

“We have established a Neural Network Noise Reduction function that enables clear, high-quality images to be obtained,” Canon explains.

“In high-sensitivity photography, it is now possible to amplify light information while simultaneously removing amplified noise and expressing smooth skin texture (skin tone), which was hindered by the occurrence of noise.”

Canon says that capturing images using current sensor technology has fundamental problems.

“In a digital camera, each pixel in the image sensor is arranged in a regular pattern, so when you shoot stripes or checkered patterns, the regular patterns are superimposed, creating moire patterns that should not exist,” the company says.

“There was a fundamental problem with doing so. In addition, each pixel of the image sensor detects only one of the three primary colors of light, red (R), green (G), and blue (B), and refers to information from surrounding pixels for the remaining two colors. RGB data for one pixel is generated by estimating the color of the pixel (this is called color interpolation processing).”

Because of this, Canon says it was hard or impossible to stop false colors or color moire from occurring, but there were processes that could suppress it at the cost of image degradation — resolution and color reproducibility would deteriorate. But thanks to the deep learning technology it developed, Canon says that it has a new processing capability for color interpolation called Neural Network Demosiac.

“Erroneous interpolation is suppressed by focusing on subjects that are difficult to guess in color interpolation processing,” Canon says.

“False colors on striped shirts, jaggies that often stand out in oblique lines, and moire and false colors that stand out in pet photos can be accurately interpolated to improve resolution and color reproducibility.”

Canon says that its deep learning technology can “strongly suppress” false colors that occur when shooting striped shirts, for example.

Finally, Canon’s neural network deep learning is able to correct for problems that occur due to issues with lenses. Even with high-quality optics that use multiple concave and convex lenses, aspherical lenses, and special materials, Canon says that it is impossible to correct for all aberrations completely. To make up for whatever can’t be corrected, its Neural Network lens Optimizer is able to correct these image degradations and can greatly improve how much resolution an image appears to have.

“Using simulation technology based on the design values ​​for each lens, we created a large number of high-precision pairs of student images with aberrations and diffraction blur and teacher images without them,” Canon explains.

“Using these as training data for deep learning, we succeeded in correcting various blurring that occurs in details when we want to clearly depict the periphery of an image, such as when shooting landscapes or astronomical objects. In addition, it is now possible to correct only the bokeh without increasing the noise that was amplified when correcting the bokeh.”

Aberration diffraction correction can also remove color fringing. On the left is the original image. On the right is the corrected image. This technology can also fix blurring that can occur at edges of a frame, especially with photos of the night sky. While chromatic aberration is one of the easiest problems to fix, Canon’s neural network also addresses this problem. “By combining the three areas of image correction realized by deep learning image processing technology, it is possible to perform a higher level of correction than by performing them individually, achieving a significant improvement in image quality such as the depiction of details and the three-dimensional effect of photographs,” Canon says. Canon doesn’t clarify if this technology will have to be applied to images after they have already been shot on a computer or if it has plans to build this technology into its cameras. It also doesn’t specify when photographers might have access to the technology. Canon’s full explanation of its deep learning technology can be found on its website, though the entire piece is in Japanese and, at the time of publication, not available in any other language

Image credits: Canon