An immersive experience

Published on 20/06/2022

Nathan Ornick is the epitome of a digital nomad. Versed in a flurry of software and creative arts, the US native is currently living between Basel and Amsterdam and working for Basel-based iart to help create a top-notch audio system, including custom headphones and a smartphone de­­vice which will carry the sound waves of the Wonders of Medicine exhibition in the Novartis Pavillon.

During his eclectic career, the 43-year-old has lent his skills to game studios, advertising agencies, artists, designers and technology companies. For this gig, he is leading a multidisciplinary team – which includes developers, engineers, designers and creatives – that will be instrumental in orchestrating the audio element of the show. It is a make-or-break endeavor because the Wonders of Medicine exhibition will offer most of its crucial content via audio. While video and gaming will play a key role too, the actual information of the show will be transmitted predominantly via headphones.

Audio guides for art and other exhibitions are typically unspectacular. They have existed for decades, ever since headphones became pedestrian items in the 1980s when Sony’s Walkman defined street coolness. Today, Apple’s elegant cordless AirPods are the latest fad.

But irrespective of the design, the real attraction of headphones is their ability to create an immersive experience, which helps users concentrate and stay focused. Following this logic, iart and Stuttgart-based Atelier Brueckner, which helped design the Wonders of Medicine exhibit, envisaged creating an experience in which people would wear headphones throughout the entire exhibition.

In contrast to conventional audio elements, where visitors press a button to hear more information about a piece of art, for example, here the idea was to create a seamless and immersive experience. This means that every time a visitor nears an object or a station during their exploration of the Wonders of Medicine exhibition, which consists of different audio and video stations that explain evolution and medicine, the audio content starts automatically.

Twin exhibition

And that is the rub. What sounds so easy and almost natural required some heavy technical lifting: “To create such an experience, we needed a complex hardware and software system, which is absolutely intuitive for the visitors to use,” Ornick explains.

Since there were no precursor projects that offered the needed flexibility and modularity, iart built the system itself. “At first, we considered some commercial solutions for parts of the system, but a lot of features in these solutions were either not needed or were missing completely. So we built our own thing to have better control and added our own magic sauce.”

The magic sauce is what Ornick calls the Mixed Reality Platform, iart’s technol­ogy framework which combines augmented and virtual reality elements and interacts with real physical things in space, all elegantly encapsulated in stylish headphones and a smartphone, which will hold the content.

To turn it around, Ornick and his colleagues treated the exhibition like a video game and created a digital twin of the Pavillon. “We built a 3D architectural space of the Pavillon. And within that architectural space we built game logic to define what happens where and when,” Ornick explains.

So, if a visitor is in a certain area, like a player in a game, the system triggers a command. But the new system can also handle more complex interactions, like when a visitor has completed a series of actions such as A, B and C, then action D is automatically triggered. “If, for example, you have watched a film to the end,” Nathan details, “our system unlocks some deeper content for you automatically.”

The overall system consists of multiple elements, including a standard Content Management System, which is stored on a server and holds the entire content of the exhibition, including video and audio. However, to reduce data transfer complexity, the audio content is pushed to a smartphone, which, together with the headphones, will be handed out to visitors before they enter the exhibition.

The content thus sits on the device, requiring no streaming because the smartphone is connected via USB to the state-of-the-art headphones, which iart developed in collaboration with Danish headphone manufacturer AIAIAI. “All the magic of these headphones is located in the headband where the custom chips are,” says Ornick. “So, when visitors are in a position where audio should be played, the system just triggers your headphones to play that audio.”

As a continuous tracking of a visitor’s position within the exhibition is key for playing the correct audio sequence, iart has integrated 36 so-called anchors – about five-centimeter-long antennas – into the ceiling of the Pavillon’s upper floor. These anchors work like a GPS and measure the distance between the visitor and the antennas.

By communicating with the chips inside the headphones, the system not only knows where a visitor is but also where they are looking since the headphones can register the orientation of a visitor’s head. Any time you walk up to an installation, the correct audio is played in the language a visitor chooses. “Whether you follow the exhibition in German, French or English, everyone can watch the same screen but have the audio in their own language.”

The idea for this new and revolutionary concept with its inbuilt flexibility was initially conceived by Valentin Spiess, the founder and chairman of iart, whose more than 20-year quest in the exhibition space has been to increasingly personalize museum and art show experiences.

The great breakthrough in the Pavillon exhibition for Spiess is the possibility to collect data to better understand how people interact with the content. “With this system, we can get very sophisticated analytics. We will be able to get some really deep insights into the audience, which previously was not possible,” Spiess says.

In the past, the success of a show was usually only judged by the number of people who visited the exhibition. “Now, we can learn a lot about a visitor’s behavior, and this helps to improve the experience over time as we can adapt the show to the needs of the audience.”

This new way of setting up exhibitions holds many opportunities for visitors and exhibition managers alike. In the past, most of the budget was spent on installing the exhibition and, once it was set up, it remained unchanged until the end. “But with our new systems, this is definitely a thing of the past now. Day one of an exhibition is the day you start to learn and adapt things over the duration of an exhibition.”

While the Pavillon exhibition has yet to open, Spiess and Ornick, who also worked on a soundscape experience for the Camille Pissarro show at the Basel Kunstmuseum recently, are already working on other tech-heavy, artificial intelligence-driven exhibition projects.

But technology is only a means to an end as Spiess explains his vision for the future: “Imagine you go to an exhibition with a friend and this friend knows everything about the exhibition and also knows you very well. This friend will provide you with the best possible exhibition experience. And that’s the goal we try to reach through technology.”

The system iart built for the Novartis Pavillon is a starting point for such a future solution, which will learn from visitor behavior and provide exhibition goers with the most interesting information. In other words, in the world of exhibitions, technology might soon replace your best friend – at least as a guide.