Apple Glasses: What's the story so far on Apple's AR headset?

For at least a decade, Apple has been working on augmented reality and virtual reality technologies.

We know this thanks to statements from the company, predictions from analysts, numerous patent filings, and Apple's interest in growing its ARKit platform for AR apps and games.

It has also hired AR/VR experts, and it's acquired multiple AR/VR companies. Add it all up, and the rumour mill suggests Apple might be developing a headset. Indeed the latest reports suggest there will be a VR headset to whet our appetites before following this up with an AR headset. 

Apple supposedly has a secret research unit comprised of hundreds of employees who are working on multiple virtual and augmented reality headset prototypes. Here's everything you need to know. 

When will Apple Glasses arrive?

It's hard to pinpoint exactly when Apple Glasses will arrive due to conflicting reports. Many predictions so far still point to 2022. In early 2021, reports suggested that the company is set to enter the "second phase of development". Around the same time it emerged, via a detailed Bloomberg report, that Apple will most likely introduce a niche VR headset before a pair of AR glasses appear. This could be as early as 2022. 

The source Bloomberg spoke to says the VR headset is codenamed N301, with the AR glasses being N421. The glasses are, according to the source, an "architecture" which Bloomberg says means Apple is still looking at underlying technologies and so would be with us by 2023 at the very earliest. Bloomberg also says development slowed during 2020 because of the inability for all engineers being able to be in the office at the same time.

Conversely, back in July 2019, DigiTimes claimed Apple halted development on all its headsets. The team developing the prototypes was apparently dissolved and reassigned to other products. At the same time, it's been claimed Apple moved Kim Vorrath, a software executive, over to the augmented reality headset division to bring "order" to the team.

Vorrath's hiring indicates Apple has not killed its AR headset team, as previously thought. References to an AR headset were also found in iOS 13 - supposedly codenamed Garta. Code in the latest version of Xcode further indicates Apple is developing an AR headset. There are even references to codenamed test devices.

And after having previously also claimed that the headset will reportedly enter mass production by the end of 2019 and might launch in early 2020, Apple analyst Ming-Chi Kuo suggested in May 2020 that they won't launch until 2022 at the earliest. Most recently he revised that expectation to a more precise date - mid-2022. Reports in April 2022, though, indicated that 2023 is now a more likely window.

If that's left you feeling confused, you're not alone - at this point, it's anyone's guess when Apple will be launching the headset.

How will Apple Glasses or a VR headset work?

Here's everything we've heard so far about Apple Glasses and what they'll be able to do if they ever come to market...

Displays

According to the early 2021 Bloomberg source, the headset will use higher-resolution displays than those currently in use in rival headsets such as Oculus. The headset could also feature an 8K display for each eye (these rumours again resurfaced in a February report from The Information). It's untethered from a computer or smartphone and should work with both VR and AR apps.

This headset can connect to a "dedicated box" over a high-speed, short-range wireless technology called 60GHz WiGig. It features a 5-nanometer Apple processor and resembles a PC tower.

AR stereo

9to5Mac said Apple is working on stereo AR in iOS that supports a face-mounted AR experience. It's been in internal testing with two Apple devices (codenamed Luck and Franc) and a third-party device, called HoloKit, which is a cardboard AR headset kit. Stereo AR apps can reportedly work in "held mode" (normal AR) or a "worn mode" when used with an external device such as a headset.

According to the early 2021 Bloomberg source, the CPU in use is apparently more powerful than Apple's M1 chips used in new Macs, so could be the A15 coming to 2021's iPhones or an M2 that is likely to come to more powerful Pro Macs this year. The CPU and GPU will be inside the headset rather than offloaded to another device (like a Mac), which is also the model Oculus has increasingly followed. And they will have a fan for cooling, too. 

AR-only or AR/VR?

It sounds like there are two different projects at play here. Respected Apple analyst Ming-Chi Kuo expects the AR glasses to be pitched as an iPhone accessory - offloading computing, networking, etc, to the iPhone - like Google's Daydream.

In 2020, Bloomberg said Apple is developing an AR product with a display, processor, and a new "rOS" or operating system. Supposedly, rOS is based on iOS. Apple is also reportedly developing a "system-on-a-package" chip for this AR headset. It's considering using touch panels, voice activation, and head gestures for input control. Virtual rooms and 360-degree video playback are also being considered.

Other reports suggest the project could offer both support for both AR and VR apps and will support eye tracking as you'd expect. Apple analyst Ming-Chi Kuo says: "Apple's eye tracking system includes a transmitter and a receiver. The transmitting end provides one or several different wavelengths of invisible light, and the receiving end detects the change of the invisible light reflected by the eyeball, and judges the eyeball movement based on the change."

Design 

According to the early 2021 Bloomberg source the first headset will be a pro-level fabric-covered device. It was also a weighty during initial testing, but apparently, it is now the same size as an Oculus Quest. The goal with the headset will be to get developers on board in time for the launch of consumer-friendly AR glasses. 

In the middle of 2019, The Information had was reporting on the company's AR glasses noting info on the design: "The lenses use a polarized system, similar to the technology in 3D movie glasses, which create the illusion of depth using stereoscopic images, the person said. The technology is similar to that in other AR and VR devices already on the market from Microsoft, Magic Leap and Facebook…

More early 2021 details from The Information reveal extra details about the headset design of the mixed reality device. There should be an interchangeable headband, much like the AirPods Max. 

In March 2022 we got some mock-ups of what the headset might look like from LetsGoDigital, too, which give a good flavour of what you might be able to expect.

Intelligent display design

According to a patent filing, it seems that Apple is working on an adaptive display for its AR headset that will account for the user's surroundings and adapt to their eye as well. The goal here is twofold. Firstly to reduce the "dazzle and discomfort" that a user would otherwise experience when transitioning out of a virtual reality viewing experience. Secondly, to make the headset more comfortable to wear by making it easier on the eye, both during use and when you take it off too.

The proposed technology will not only measure the ambient light in the surrounding environment but also measure things like the "blink rate, pupil size, and eye openness" to adjust the headset brightness accordingly. 

As well as making it more comfortable, the proposed tech will help to "... boost the perceived dynamic range of the display." This should make the experience more satisfying, improve comfort and hopefully save battery too. 

Haptic feedback

A patent spotted in February 2021 has indicated that Apple is also looking into how it can use haptic feedback as part of its headset. The patent is applied widely to all potential devices, but given that its main other options are iPhones, tablets and computers, it would make a lot more sense to see the potential system matched up with a headset. 

It sounds like a mighty complicated idea, too, in classic Apple style. The idea is to have a motion sensor monitoring for the momentum and force of the user's movements in relation to virtual objects viewed presumably through the headset. This would be paired up with a worn haptic feedback system that could provide feedback according to how the virtual objects move. 

Apple notes that the system might be too bulky to work, and it certainly sounds like an accessory more than a core part of its headset plans, but it's an interesting potential avenue for the company to explore.

On the lenses

The person The Information spoke to said making the AR lenses is especially challenging because they are composed of multiple, extremely thin layers of different synthetic materials, each of which is susceptible to bubbles, scratches and other marks. To reduce defects, the lenses must be manufactured in dust-free zones known as clean rooms."

Apparently, the headset doesn't have space for prescription glasses according to the early 2021 Bloomberg report - instead custom lenses can be inserted into the headset. Apple is also testing the built-in cameras for hand tracking and is also said to be working on a system to input text by 'typing' in the air. 

Has Apple ever spoken about Apple Glasses?

Apple CEO Tim Cook has said the technology does not exist to create AR glasses in a quality way. "But today I can tell you the technology itself doesn't exist to do that in a quality way," he explained. "The display technology required, as well as putting enough stuff around your face - there are huge challenges with that. The field of view, the quality of the display itself, it's not there yet."

Keep in mind, in 2016, he also said: "AR can be really great... We have been and continue to invest a lot in this. We're high on AR in the long run... I think AR is big and profound. This is one of those huge things that we'll look back at and marvel at the start of it."

Who is part of the Apple Glasses team?

Apple has hired several employees with expertise in AR/VR technology, including computer science professor Doug Bowman, who led the Virginia Tech's Center for Human-Computer Interaction. He focuses on three-dimensional user interface design. Apple also hired employees that have worked on Microsoft's HoloLens team, and at Lytro, a company working on a VR camera.

Apple also hired Zeyu Li, who served as a principal computer vision engineer at Magic Leap, as a "Senior Computer Vision Algorithm Engineer." Another hire is Yury Petrov, a former research scientist at Facebook-owned Oculus, who is now serving as a "research scientist" at Apple. Augmented reality expert Jeff Norris also joined Apple in April 2017 as a senior manager. He worked at NASA.

In May 2018, Apple hired Sterling Crispin, who developed a painting app for mobile VR headsets called Cyber Paint. In December 2018, it hired former senior Tesla and Microsoft HoloLens designer Andrew Kim. There's also Jaunt VR founder Arthur van Hoff, who joined Apple as a senior architect in April 2019. Keep in mind Apple's AR glasses/headset team has hundreds of other employees.

In late June 2020 it came to light that Apple is hiring a lot of Magic Leap's employees after the failure of that AR headset. And according to Bloomberg, Apple now has over 1,000 employees working on the project. 

Which AR/VR companies has Apple bought?

Not only has Apple hired AR/VR experts, but it's also acquired companies that specialise in this area. In 2017, it purchased Vrvana, a company that developed a mixed reality headset called Totem. Around that time, it also bought Akonia Holographics, a company that makes lenses for AR smart glasses. Apple also purchased Israeli-based 3D body sensing firm PrimeSense way back in 2013.

And in 2020, Apple bought NextVR, a company which gave sports and concerts a VR platform. The company had partnered with Fox Sports, Wimbledon and the NBA among others. 

Other companies it's purchased include Metaio, Faceshift, Emotient, Flyby Media, and RealFace.

Source : https://www.pocket-lint.com/ar-vr/news/apple/136553-apple-glasses-ar-headset-what-s-the-story-so-far