skip to main content
Guernica painting
Guernica painting.

Beauty is in the ear of the beholder

Industrial and systems engineers are making art more accessible to those who cannot consume it visually through auditory augmented reality.

Pitch, timing, rhythm, tempo, and key. While these words usually come together to describe music, they can also describe paintings.

In their current form, art galleries and museums may exclude certain populations, including blind or visually impaired audiences, and even children or smaller adults who could have a hard time seeing certain exhibits if they're obstructed by groups of people. 

Myounghoon "Philart" Jeon, associate professor in the Grado Department of Industrial and Systems Engineering (ISE), and his students are working to make art more accessible to those who cannot consume it visually. They start with a painting and end with a sound file that captures the essence of the painting by blending digital and visual dimensions, in what is called auditory augmented reality. 

Auditory augmented reality means enriching the real environment - such as the paintings in an exhibit - with virtual audio components. In this case, the researchers are adding one more perceptual dimension to pieces of visual art, making what is usually a strictly visual experience, one that those with visual impairments can also enjoy. 

“Right now, augmented reality is primarily a visual modality. So we wanted to identify the parameters if we wanted to add an extra dimension, in this case, audio, to augmented reality,” said Jeon. “That's how we started this.”

Pioneering the fusion of art and technology

The research is primarily led by Abhraneil Dam, a Ph.D. student in ISE, and takes place in the Mind Music Machine (tri-M) Lab, a transdisciplinary research group based in ISE, computer science, and Human-Centered Design programs at Virginia Tech. This project was supported by the Institute for Creativity, Arts, and Technology (ICAT) as a Mini grant.

The team’s research focuses on sonifying, which means translating data into sound, so that people can “hear” the paintings. A variety of algorithms and software are used to create the audio for each painting, and the process can be broken down into four phases.

In phase one, the team analyzes the visual parameters of the painting. These parameters are what make up the overall artistic style of a painting and can include color, shape, texture, and shadows, to name a few. “An algorithm is used to detect the color of the painting and distribution of the colors and percentage of the hues,” explained Jeon. 

In the second phase, the visual parameters are translated into musical parameters. This is done by generating a musical instrument digital interface (MIDI) file. A MIDI file is not the sound itself, but is similar to an Excel file that contains all the musical parameters, including pitch, timing, rhythm, tempo, and key. “This is kind of a blueprint of the sound,” said Jeon.

In the third phase, the team enhances the blueprint of the sound to make it consistent and ensure it accurately represents the genre of the painting. Using music software known as Ableton Live, the team designs an advanced music piece based on the MIDI file. 

Abhraneil Dam in a lab.
Abhraneil Dam demonstrates how the technology analyzes a painting. Photo by Peter Means.
Yeaji Lee works on a computer.
Yeaji Lee uses Ableton Live, audio editing software, to enhance the audio. Photo by Peter Means.
A computer screen showing audio editing software.
Ableton Live editing environment. Photo by Peter Means.

“We can play the MIDI file itself so you can hear the audio, but it could sound differently depending on the computer system used,” said Jeon. “It sounds different depending on the computer. By creating an actual sound file, like a .wav file, for example, it will sound exactly the same way across systems.”

When creating the sound file, the goal is to reflect the genre of the painting, whether that is abstractionism or realism or romanticism.

“For example, for abstractionism we use electronic instruments, and for realism we use more classical instruments,” explained Jeon. If the painting is brighter overall, a major scale is used. If it's darker, a minor scale. “If it's impressionism, we want to express something like a dreamy-type sound using soft warm instruments.”

For Yeaji Lee, Ph.D. student in industrial and systems engineering, sounds can amplify an artist’s impact by “enhancing the stories or narratives people envision when viewing a painting. Augmenting their understanding of the artwork through sound is what defines auditory augmented reality.”

Listen to the audio files below to hear the difference between the sonified MIDI file and the enhanced file:

The Return (1837) by Thomas Cole

The Return painting

Play the audio file on the left to hear the computer-generated music for the painting above. Play the audio file on the right to hear the enhanced music created by the team.

Water-Lily Pond (1899) by Claude Monet

Water Lilies painting.

Play the audio file on the left to hear the computer-generated music for the painting above. Play the audio file on the right to hear the enhanced music created by the team.

Demonstrations make up the fourth phase of the research process. The team uses bone conduction headphones to showcase this type of technology, which, instead of covering the ear and obstructing all sound, vibrates the bone. The bone vibrations then deliver the sound directly into the inner ear. 

“When blind people use assistive devices like navigation, typically they use bone conduction because they can also hear the outside sound for safety reasons,” said Jeon. “In a museum setting, they can use bone conduction headphones so they can hear the environmental sound, while also listening to the sounds of the painting.”

Bone conduction headphones.
Bone conduction headphones. Photo by Peter Means.

After demonstrating paintings with different sounds to the audience, the team collected their reflections and did some sentiment analysis, which is a way of gauging emotional responses. Based on the transcription of their qualitative comments, they identified different types of emotions from that transcription.

Most people, when asked to guess which sound went with which painting, guessed correctly. “That means we did a great job, and tells us that we can continue in this direction,” said Jeon.

Expanding access to art beyond the museum

Jeon hopes to unlock the magic of art for those outside of a gallery setting. He’s even chosen a nickname for himself that represents his passion for art.

“The nickname Philart means ‘love art’. I love art and I’m always interested in how we can make a playground for artists and musicians," he said.

While this technology can be used in art galleries, it could also provide an immersive experience of art in the virtual reality (VR) environment, a computer-generated environment with scenes and objects that appear to be real, making the user feel they are immersed in their surroundings. People who cannot visit museums or art galleries in person have the option to use virtual reality technology at home. By wearing VR goggles, they can have a sensory experience of art pieces, encompassing both visual and auditory elements. 

Lee hopes this research will help people from all walks of life enjoy and experience artwork in new and innovative ways – and expand their understanding of visual art through sound.

“It’s wonderful to realize that someone else feels exactly the way I do when I look at a painting,” she said. “It’s a great feeling to know I can help them understand a visual piece through sounds I create.”

Yeaji Lee, Abhraneil Dam, and Myounghoon "Philart" Jeon.
From left to right: Yeaji Lee, Abhraneil Dam, and Myounghoon "Philart" Jeon. Photo by Peter Means.

If you want to have an impact on our students and faculty like those featured in this magazine, go here to support the College of Engineering. For more information, call (540) 231-3628.