Beauty is in the ear of the beholder

Pitch, timing, rhythm, tempo, and key. While these words usually come together to describe music, they can also describe paintings.

In their current form, art galleries and museums may exclude certain populations, including blind or visually impaired audiences, and even children or smaller adults who could have a hard time seeing certain exhibits if they're obstructed by groups of people. 

Myounghoon "Philart" Jeon, associate professor in the Grado Department of Industrial and Systems Engineering (ISE), and his students are working to make art more accessible to those who cannot consume it visually. They start with a painting and end with a sound file that captures the essence of the painting by blending digital and visual dimensions, in what is called auditory augmented reality. 

Auditory augmented reality means enriching the real environment - such as the paintings in an exhibit - with virtual audio components. In this case, the researchers are adding one more perceptual dimension to pieces of visual art, making what is usually a strictly visual experience, one that those with visual impairments can also enjoy. 

“Right now, augmented reality is primarily a visual modality. So we wanted to identify the parameters if we wanted to add an extra dimension, in this case, audio, to augmented reality,” said Jeon. “That's how we started this.”

Pioneering the fusion of art and technology

The research is primarily led by Abhraneil Dam, a Ph.D. student in ISE, and takes place in the Mind Music Machine (tri-M) Lab, a transdisciplinary research group based in ISE, computer science, and Human-Centered Design programs at Virginia Tech. This project was supported by the Institute for Creativity, Arts, and Technology (ICAT) as a Mini grant.

The team’s research focuses on sonifying, which means translating data into sound, so that people can “hear” the paintings. A variety of algorithms and software are used to create the audio for each painting, and the process can be broken down into four phases.

In phase one, the team analyzes the visual parameters of the painting. These parameters are what make up the overall artistic style of a painting and can include color, shape, texture, and shadows, to name a few. “An algorithm is used to detect the color of the painting and distribution of the colors and percentage of the hues,” explained Jeon. 

In the second phase, the visual parameters are translated into musical parameters. This is done by generating a musical instrument digital interface (MIDI) file. A MIDI file is not the sound itself, but is similar to an Excel file that contains all the musical parameters, including pitch, timing, rhythm, tempo, and key. “This is kind of a blueprint of the sound,” said Jeon.

In the third phase, the team enhances the blueprint of the sound to make it consistent and ensure it accurately represents the genre of the painting. Using music software known as Ableton Live, the team designs an advanced music piece based on the MIDI file. 

“We can play the MIDI file itself so you can hear the audio, but it could sound differently depending on the computer system used,” said Jeon. “It sounds different depending on the computer. By creating an actual sound file, like a .wav file, for example, it will sound exactly the same way across systems.”

When creating the sound file, the goal is to reflect the genre of the painting, whether that is abstractionism or realism or romanticism.

“For example, for abstractionism we use electronic instruments, and for realism we use more classical instruments,” explained Jeon. If the painting is brighter overall, a major scale is used. If it's darker, a minor scale. “If it's impressionism, we want to express something like a dreamy-type sound using soft warm instruments.”

For Yeaji Lee, Ph.D. student in industrial and systems engineering, sounds can amplify an artist’s impact by “enhancing the stories or narratives people envision when viewing a painting. Augmenting their understanding of the artwork through sound is what defines auditory augmented reality.”

Listen to the audio files below to hear the difference between the sonified MIDI file and the enhanced file:

Demonstrations make up the fourth phase of the research process. The team uses bone conduction headphones to showcase this type of technology, which, instead of covering the ear and obstructing all sound, vibrates the bone. The bone vibrations then deliver the sound directly into the inner ear. 

“When blind people use assistive devices like navigation, typically they use bone conduction because they can also hear the outside sound for safety reasons,” said Jeon. “In a museum setting, they can use bone conduction headphones so they can hear the environmental sound, while also listening to the sounds of the painting.”

After demonstrating paintings with different sounds to the audience, the team collected their reflections and did some sentiment analysis, which is a way of gauging emotional responses. Based on the transcription of their qualitative comments, they identified different types of emotions from that transcription.

Most people, when asked to guess which sound went with which painting, guessed correctly. “That means we did a great job, and tells us that we can continue in this direction,” said Jeon.