AR/VR Needs Disruptive, Smart Imaging Systems

Vision is so important to humans that almost half of your brain’s capacity is dedicated to visual perception. It’s no surprise that hyperscalers such as Meta, Microsoft, and Apple have bet on augmented/virtual reality (AR/VR) – starting with augmented vision – to become the new machine-human interface. Since our eyes are the second most complex organ after our brains, it makes sense for AR/VR to replace personal computers and smartphones. However, many technical hurdles remain in electronics (the brain) and optics (the eyes). In my first blog post on trends in imaging design, I discussed how digital twins will foster mass customization and data optimization through end-to-end simulation of imaging systems, from manufacturing and testing to user experience virtualization. In this blog post, I will focus on how AR/VR is driving innovation in imaging design. Whoever thought that digital twins include only mechanical, thermal, or electrical components are missing the need for AR/VR systems to develop disruptive, optical, smart imaging systems.

Imaging Systems Go Digital

Similar to electronics several decades ago, today’s optics are primarily analog. Optical systems consist of complex bulky surfaces, stacks of thin films, or continuous shapes. It’s only recently that an alternative to this traditional approach arose. Flat optics emerged to create nanostructured devices (with features smaller than a wavelength) able to replicate optical functions such as focusing, color filtering, and reflecting. The main advantage of this technology is its compactness, versatility, and compatibility with semiconductor processes, unleashing new possibilities for optics and electronics co-integration. Many hurdles need to be overcome to bring this technology to maturity. Leaders in materials and semiconductors processes, such as Applied Materials, are using their expertise to help drive flat optics technology into various volume markets.

Every optical element will transition from analog to digital. Fully digital imaging systems would collect or display images in a more agile and specific way. Foveated cameras or displays are an example of how pixelated imaging systems could be used in a more customized and efficient manner, bringing detailed visual information precisely to where it has to be processed and leaving other areas with less resolution. In a sense, digital imaging systems will mimic how our eyes function.

Designing for the Future: Imaging Systems Go Smart

Half of our brain processes visual information. Similarly, imaging systems have to become smart, which means optics and electronics must be co-designed, co-optimized, and maybe even co-manufactured.

An imaging system is composed of a sensor that collects the light, a converter that transfers the raw sensor signal to a processing unit, and a computer vision unit that post-processes the digital signal to make a decision. Each component is designed separately by different teams and manufactured on various technologies.

Near or in-sensor computing is a major trend towards more innovative imaging systems with optimized energy consumption, low latency, and improved security. For example, CMOS image sensors could consist of a planar System on Chip that would integrate a photodiode array, pulse-modulated circuits, and a simple ADC converter to have a front-end processing unit near or into the sensor layer. Other  approaches could include a solution in which the optics, the sensor, and the algorithms are jointly optimized and then used in tandem with one another as a solution to a unique imaging problem: recognizing an object to make a decision is different from taking souvenir pictures, video surveillance cameras should be different from eye-tracking ones. Display engines could also become so smart and fully optimized that we might not even need to build intermediate images to render truly immersive content .

The next generation of imaging systems promises to extend AR/VR technology so that it can replace personal computers and smartphones. It is an exciting time for the optics industry and especially for the catalyst ecosystem that is the design of smart imaging systems.

In Case You Missed It

Catch up on our other recent optical blog posts:

• Structural, Thermal and Optical Performance (STOP) Analysis in CODE V
• LightTools Features Drive Efficient Illumination Optics Design
• How to Reduce Stray Light in Your Optical Design Project with Scattering Measurements

Sources:

• ¹A compact camera module is $6 for a smartphone: Technical innovations and market demand are shaping the Compact Camera Module industry – System Plus Consulting
• ¹The lens set market is 1/7th of the global compact camera module market: Status of the Camera Module Industry 2019 – Focus on Wafer Level Optics – Yole Développement (systemplus.fr)