My process of designing a feature that translates music into visuals for the deaf and hard of hearing
In my last year studying digital design at FIT, I got an assignment to design anything under one condition: it had to be digital. Any creative knows that more freedom equals difficulty.
And so I was frustrated scrolling through the world of the Internet for an interesting topic that I also cared about, until I watched this:
This video features Amber Galloway Gallego, an American Sign Language(ASL) interpreter, who translates all the elements of a song (lyrics, instruments, beats, and most importantly, the feel) into ASL. Each ASL music interpreter does so differently, making the music-listening experience more fun and visceral for the deaf and hard of hearing.
After the video ended, I googled right away and found out that there are 466 million people in the world with a disabling hearing loss. Instantly, I had a problem and an insight.
Insight — it’s challenging for the deaf and hard of hearing to enjoy and understand songs.
But I didn’t know where to begin. I didn’t personally know anyone who is deaf or hard of hearing nor had I ever met one in my life. All I knew was that I wanted to design something that would translate songs into visuals but had a limited time frame and resources. So I started with what I could learn online.
I’d like to walk you through my story in three parts which are:
One of my goals for this project was to influence the daily music-listening experience of my target audience. I hoped to make the experience more fun and less discriminative so that people immerse themselves whenever they want to, wherever they are. To achieve this, it was imperative that I design for mobile; so I first checked what’s currently out there.
There is an overflowing number of music apps. But for the deaf community? Only a few exist. I was disappointed in this noninclusive reality and became more assertive about the need of a solution. I believed that the apps that already exist have room for improvement in both experience and visual design.
Additionally, I learned that some apps can examine your hearing ability by playing tones that range from low to high and asking you to press the button whenever you hear the tone. By providing detailed results about both ears, they inform the users about their own hearing ability. This inspired me to incorporate a similar function into my solution.
To verify that visuals actually help people understand the context of music, I conducted a survey, asking people if they think they’ll feel more connected to music if they visually saw it. I asked 25 people and:
76% said “yes”, 16% said “maybe”
However, there were two big problems with my survey that made the results unreliable and unrepresentative of my target audience:
- The sample size was small.
- Majority of the participants were hearing.
I had to find a way to talk to my actual target audience. I visited one of the ASL schools here in New York with a hope to talk to the students but I needed to go through some steps (i.e. talk to the principal, the teachers, etc.).
Although I’d have loved to, the time I had for this project didn’t allow me to. That’s why I looked for a different resource that I had access to all the time (maybe too much access).
I turned around to Instagram to interview my target audience.
Using hashtags like #deaf and #deafcommunity, I found people who were deaf, hard of hearing or ASL music interpreters. I reached out to them through Direct Message to ask questions and found insights that basically led my project.
The following is a few examples of the DM interviews.
Christopher | Hard of hearing
- “I listen to music every single day. It plays a huge role in my life. It’s one of the few things that keeps me in my zone.”
- “The videos do help understand what the music is exactly about than my imagination.”
Lea (ASL Music Interpreter) | Hearing
- “The most important thing is to understand and interpret the context of a song.”
- “I think that everyone has their own way to interpret music.”
Raquel | Hard of hearing at birth, currently deaf
- “If the music isn’t playing on my iPhone, it’s definitely the vibrations.”
- “But if there are no lyrics, I love the vibrations of the instruments.
Without the DM interviews, it would have been impossible for me to learn about the following 6 insights, which emerged most frequently throughout my research:
- Understanding the context of the song is key.
- Everyone interprets songs differently.
- Some people may not know sign language.
- Videos with lyrics help build connections.
- Some people can hear certain pitches.
- Vibrations play a crucial role.
Introducing the Apple Music Visualizer, a new feature in Apple Music (that I wish Apple actually built) that translates music into visuals for the deaf and hard of hearing. Let me walk you through my design process.
Now that I had an understanding of my target audience, I could dive into userflows and really put myself into their shoes.
1. Enable/disable in Settings
Apple already has its own system for accessibility. With a simple addition to the hearing section under Accessibility, users can easily enable Apple Music Visualizer.
II. Inside Apple Music
Apple Music primarily has 5 sections — Library, For You, Browse, Radio, and Search. The ‘For You’ has recommendations and your recent playlists and the hearing ability test lives inside this section.
After the user takes the test, the original ‘For You’ features will return. Users can also change the visualizer settings if they’d like to, under ‘Visualizer Settings’.
3. Finally, the Features
Following Apple’s official design principles and paying closer attention not to distract the original system, I designed the four main features.
During the DM interviews, I was shocked to learn that not everyone who is deaf and hard of hearing knows how to sign. It totally makes sense because it’s up to you to learn a whole new language- but I was surprised at my own ignorance. That’s why I added an option to ‘Watch the Lyrics,’ especially for those who aren’t familiar with ASL.
The biggest challenge I faced was thinking about improvements without making it too different. Because why should they be, right?
I wore different hats while working on this project over the course of 3 months. What seemed to be just another design project turned out to be the most exciting and memorable one for me, changing my perspective towards UX/UI design and guiding me further into this industry.
Fun fact: when I got back to my interviewees with the completed project, they told me that they wish they had something like this. Some even thought it was actually a real feature Apple Music has launched.
I so wish this kind of feature could come to life. And I hope my small act of designing inspires others to consider accessibility and inclusiveness in their daily designer lives, too.
Thanks for reading!
Bringing music to the deaf by designing a new feature in Apple Music was originally published in UX Planet on Medium, where people are continuing the conversation by highlighting and responding to this story.