U.S. Army to use HoloLens technology in high-tech headsets for soldiers

HoloLens technology in high-tech headsets for soldiers

Hololens technology In 2019, as part of a new effort to develop advanced technologies to enhance situational awareness and capabilities for soldiers, the U.S. Army invited Microsoft engineers and other employees to a military base in North Carolina for a series of weeklong mini boot camps.

The Microsoft team learned about navigating on the ground, maneuvering in the dark and communicating in the field. The goal was to help the engineers understand the challenging environments soldiers operate in, then use that knowledge to design a mixed-reality headset that will soon be rolled out to thousands of U.S. troops.

“Because Microsoft was a non-traditional defense contractor, we felt like for them to be able to experience what soldiers actually feel and what they go through would help with making some of their technical decisions,” says Mark Stephens, the Army’s deputy program manager for the project.

“Some folks told me it was a pretty eye-opening experience.”

The exercise exemplified a unique partnership between Microsoft and the U.S. Army, which recently awarded the company a contract to produce mixed-reality headsets based on Microsoft’s HoloLens technology. The devices, using what is called the Integrated Visual Augmentation System (IVAS), will allow soldiers to see through smoke and around corners, use holographic imagery for training and have 3D terrain maps projected onto their field of vision at the click of a button.

Photo of soldier standing in front of a military vehicle and wearing a mixed-reality headset.

Soldiers provided feedback that shaped many of the headsets’ key features. (Photo by Courtney Bacon, U.S. Army)
The Army plans to start equipping soldiers with the headsets in September, and leaders say the devices will fundamentally change how they operate and what they can do.

“Soldiers will be able to rehearse and train in more realistic scenarios using augmented reality to prepare themselves for what they are walking into,” says Master Sgt. Marc Krugh, a senior enlisted advisor who’s worked on the project since its inception.

“Inevitably, IVAS is going to save lives,” Krugh says. “That’s our main focus – bringing our men and women in arms back home.”

The technology was developed through a collaborative process that brought Microsoft’s user-centric design approach to the military for the first time. Microsoft worked closely with the Army in an integrated team from the start to better understand its needs and goals for the project. Soldiers came to Microsoft’s industrial design and software labs at the company’s Redmond, Washington, campus to test various prototypes and provide feedback, which was then used to develop the next version of the device.

By February 2021, the team had collected almost 80,000 hours of soldier feedback, including four rounds of testing within a six-month period during the COVID-19 pandemic. The group used Microsoft Teams to meet and share documents and Microsoft’s Power BI data visualization tool to track the status of the project.

“The whole process was really iterative,” Krugh says. “Using the prototyping process, we were able to find out what works and what doesn’t work. Had we not received feedback from soldiers, we might have done the wrong thing and developed in the wrong direction.”

Microsoft Technical Fellow Alex Kipman, who created HoloLens and its precursor, the Kinect motion-sensing line of devices, says the Army’s level of involvement made the project unique.

“It’s unusual for the government to have gone on this journey with us,” Kipman says. “I give infinite credit to the Army for their level of growth mindset. It was the right people with the right attitude at the right time to take the leap of faith with us.”

Soldier input directly shaped many of the headset’s key features. Early prototypes included a HoloLens headset that engineers had affixed cameras to, goggles with weights attached, and various 3D-printed devices. The devices met the Army’s requirements of rugged, waterproof and shockproof. But no one had considered, for example, that the headsets needed to allow users to brace a rifle against a cheek.

“If a soldier wanted to put his or her cheek on the butt of their weapon, if there were sensors on the bottom of the headset, they wouldn’t be able to do that,” Krugh says.

Feedback from soldiers also identified that the rim of the goggles needed to be redesigned for better peripheral vision. Dozens of prototypes were created, tested and refined.

“That allowed us to iterate and change and adapt and overcome issues that a lot of programs in the past would not have been able to do,” Krugh says. “Microsoft has been very good with soldier feedback, following it and identifying some things that just won’t work and then finding a different way to tackle those specific issues.”

Photo of two soliders wearing mixed-reality headsets.

Microsoft worked closely with the Army in an integrated team to understand its goals and needs for the project.

Microsoft initially designed a dial for the compute pack – known as the “puck” – that soldiers wear on their chests to control the headsets, with the thought that it would be easy to manipulate. But when soldiers put the pack on and crawled on the ground, the carefully designed dials promptly broke off.

“We designed this beautiful dial that you can use with one finger and I’m super proud of it,” Kipman says, “and if we shipped the product with it, we would have passed the spec, and they would have broken and not worked for many soldiers.

“There are literally thousands of examples like that throughout this program, and it’s why we do what we do,” he says.

IVAS uses HoloLens’ mixed-reality technology plus thermal imagery, sensors, GPS technology and night vision capabilities to improve soldiers’ situational awareness and give them critical information to help plan, train and carry out missions. The devices enable soldiers to see where they are and what’s around them by projecting holographic images, three-dimensional terrain maps and a compass onto their field of vision.

The device represents a paradigm shift for the Army, both in the way it was developed and what it will enable soldiers to do. Instead of planning missions with terrain models cobbled together with boxes, sticks, rocks and other improvised materials, IVAS enables soldiers to use 3D maps depicting the places they will be.

Soldiers could, for example, use IVAS in the future to shift their view to the vantage point of combatants who might see them approaching or see what a particular environment looks like from different directions or in daylight, moonlight or rain. They could view a holographic image of a building and determine how best to access it or escape.

Closeup photo of a soldier wearing a mixed-reality headset.
Soldiers have been testing the prototypes for two years.
Since IVAS uses personnel location information that is distributed over a tactical network, soldiers will be able to see where their other platoon members are, even in dark or dense environments. Thermal technology will allow soldiers to see through smoke, and the system features improved night vision capabilities.

“Think about giving the soldier instantaneous situational awareness, not only of their surroundings, but also of the proximity of mission-critical people, places and things. This will have a profound impact on soldier safety and a marked reduction in friendly fire and other types of incidents,” says David Marra, IVAS program director for Microsoft.

When IVAS is paired with a weapon sight – an aiming device – it also enables soldiers to see what the scope on their rifle sees, allowing them to safely look around corners. Technology transmits what the scope sees to the soldier’s headset, regardless of which direction the soldier is looking, and soldiers can communicate digitally between the headsets.

Though the devices are still being tested by the Army, IVAS has already changed how soldiers operate, Krugh says.

“The capabilities that we have at this point in the program have allowed soldiers to rethink how they do things,” he says. “This is going to put a new spin on things and it’s going to ultimately mean a more knowledgeable and prepared force.”

IVAS is part of a broader effort to modernize U.S. military operations and maintain the nation’s leadership in advanced technologies. In 2018, Microsoft won a $480 million contract from the Army to develop a mixed-reality headset to help soldiers train, rehearse and fight.

The device would use HoloLens technology, but for a whole new purpose. Nothing like it existed. Microsoft has long employed a user-centric approach to create new products and understood that due to the pace of the project, that approach would be crucial to its success.

The Army adopted the method to use feedback from soldiers to develop and test the device at each stage, including four major “soldier touchpoints,” or testing rounds, and multiple user studies in between. While the Army has previously included soldiers in its equipment development process, “soldier-centered design” was a departure from the usual way of doing business.

“From a military perspective, we tend to have a habit of going back into the files, grabbing something that looks, smells and feels like what somebody asked us to do and try to recreate something or shift off of a known point,” says Maj. Gen. Anthony Potts, the Army’s program executive officer over the project.

“With IVAS, we absolutely made the decision not to do that.”

Soldiers have been testing the prototypes at Fort Pickett, Virginia, for two years. More recently, the devices were tested in Puerto Rico and Alaska to ensure they perform well in tropical and cold environments.

Under the new contract, which could be worth up to $21.88 billion, Microsoft will initially produce more than 120,000 headsets for soldiers at a Silicon Valley manufacturing facility. The five-year agreement can be extended for another five years. The devices will first be used by soldiers on foot, and the Army is also conducting experiments with using IVAS in military vehicles so soldiers can see what’s around them before stepping outside.

IVAS is being developed under an agreement known as an Other Transaction Authority, which gives the Army more flexibility to rapidly develop and test new technology in collaboration with non-traditional defense contractors. The arrangement enabled IVAS to be developed in less than three years, much faster than a traditional project of its type.

But Krugh says the project’s success to date also depended on having the right partner. The Army had an idea of what it wanted when it launched the initiative, “but it took a company like Microsoft to really identify what that looks like,” he says.

“Microsoft has done a really good job of trying to understand what the government needed and also to add some things into it that we didn’t know we needed or didn’t even know were in the realm of possibility.”

Top photo: A soldier tests an IVAS headset during a training exercise at Fort Bragg, North Carolina. Photo by Courtney Bacon, U.S. Army. Other photos courtesy of the U.S. Army.

Source: Microsoft News
Deborah Bach
Jun 8, 2021