Designing a Generative AI-Assisted Music Psychotherapy Tool for Deaf and Hard-of-Hearing Individuals

This paper presents a co-designed generative AI tool that enables Deaf and Hard-of-Hearing individuals to engage in music psychotherapy through visual and conversational songwriting, demonstrating that collaborative human-AI interaction can effectively facilitate emotional release and self-understanding for this underserved population.

The Gist

Imagine music as a universal language of the heart. For most people, singing a song or humming a tune is like opening a window to let fresh air (emotions) flow in and out. But for people who are Deaf or Hard-of-Hearing (DHH), that window is often locked. Traditional music therapy relies heavily on hearing the sound, which can leave DHH individuals feeling excluded from a powerful tool for healing.

This paper is about building a new kind of bridge to that window. The researchers created a digital tool that uses Artificial Intelligence (AI) to help DHH people write their own songs, not to hear them perfectly, but to feel and see their emotions.

Here is the story of how they did it, explained simply:

1. The Problem: The "Silent" Gap

Think of music therapy like a dance class. If the instructor only speaks and you can't hear them, you might feel lost. The researchers found that while DHH people often use hearing aids or cochlear implants to hear sounds, they still struggle with the emotional side of music. They often get therapy focused on "fixing" their hearing, rather than using music to heal their hearts. They needed a way to express their feelings without needing perfect ears.

2. The Solution: A "Digital Co-Pilot"

The team built a web-based tool that acts like a creative co-pilot. It combines two types of AI:

• A Chatbot Therapist: A friendly AI conversation partner that asks questions and listens without judging.
• A Music Generator: An AI that turns your words into actual songs.

But here is the magic: The tool speaks the language of the DHH community. Instead of asking, "What does this melody sound like?" (which is hard if you can't hear well), it asks, "What color is your sadness?" or "If your frustration were a storm, what would the sky look like?"

3. How It Works: The Four-Step Journey

The researchers worked closely with real music therapists to design a four-step process, like a guided tour through your own mind:

• Step 1: The Warm-Up (Building Trust): The AI chatbot starts a gentle conversation. It's like sitting down with a friend who never interrupts. It uses empathy (saying things like, "That sounds really heavy") and offers multiple-choice answers so users don't feel stuck trying to find the perfect words.
• Step 2: Painting with Words (Lyrics): Instead of forcing users to write poetry, the AI helps them visualize. If a user is sad, the AI might ask, "Is your sadness a gray rainy day or a heavy blanket?" This turns abstract feelings into concrete pictures that can be turned into lyrics.
• Step 3: The Soundtrack (Making Music): Once the lyrics are ready, the AI generates a song. But the user doesn't just listen; they watch. The tool turns the music into a visual show—lyrics dancing on the screen, colors changing with the mood, and shapes pulsing with the beat. It's like watching a movie of your own feelings.
• Step 4: The Reflection (Looking Back): After the song is made, the AI asks, "How does this song make you feel now?" This helps the user realize they have processed their emotions, often leading to a sense of relief or a new perspective.

4. What Happened in the Test?

The researchers tested this tool with 23 DHH adults. The results were like watching a garden bloom:

• The "Safe Space" Effect: Many participants said they felt safer talking to the AI than to a human. Why? Because the AI doesn't look at them, judge their voice, or get impatient. It's a judgment-free zone.
• The "Visual" Key: The visual metaphors were the secret sauce. One participant described their divorce as "sitting by a rainy window." The AI turned that image into a song. Another person described job-hunting stress as "rough waves calming down."
• The "Aha!" Moment: When users heard their own stories turned into music, many felt a wave of relief. It was like taking a heavy backpack off their shoulders. Even if the AI made a song that wasn't exactly what they imagined, users often found a new meaning in it, realizing, "Oh, I'm actually more anxious than I thought," or "I'm stronger than I realized."

5. The Big Takeaway

This study proves that you don't need to hear music perfectly to use it for healing. By translating sound into visuals, stories, and AI conversations, we can open the door to music therapy for everyone.

Think of it this way: If music is a lighthouse guiding ships through a storm, this tool builds a new kind of lighthouse that uses bright lights and clear signals for those who can't hear the foghorn. It shows that with the right technology, everyone can find their own song, even in the silence.

Technical Summary

Here is a detailed technical summary of the paper "Designing a Generative AI-Assisted Music Psychotherapy Tool for Deaf and Hard-of-Hearing Individuals."

1. Problem Statement

Traditional music psychotherapy relies heavily on auditory perception, which excludes Deaf and Hard-of-Hearing (DHH) individuals from its therapeutic benefits. While DHH individuals often face social isolation and psychological distress, existing interventions for this population focus primarily on auditory rehabilitation (e.g., training residual hearing or cochlear implant perception) rather than emotional expression and healing. Furthermore, songwriting—a powerful therapeutic tool for uncovering unconscious emotions—is often inaccessible to DHH users due to:

• Auditory barriers: Difficulty engaging with sound-based creation.
• Lack of inclusive tools: Absence of systems that accommodate the visual, tactile, and narrative sensory modalities preferred by DHH individuals.
• Therapeutic gaps: A lack of remote, accessible tools that support emotional care outside of clinical settings.

The study aims to bridge this gap by designing a Generative AI (GenAI)-assisted music psychotherapy tool that enables DHH individuals to co-create songs with an AI, facilitating emotional expression and self-reflection without relying on auditory dominance.

2. Methodology

The research followed a mixed-methods approach involving three distinct phases:

Phase 1: Needs Assessment (RQ1)

• Participants: Six licensed music therapists with experience working with DHH clients.
• Method: Semi-structured individual interviews (60 mins each).
• Goal: To identify current challenges in DHH music therapy and the specific needs for supportive tools.
• Key Findings: Therapists noted that DHH clients often prioritize functional auditory training over emotional healing due to time/cost constraints. Clients frequently struggle with social isolation, low self-esteem, and a lack of confidence in sound-related activities. There is a critical need for remote, multimodal tools that do not rely on auditory perception.

Phase 2: Co-Design Workshop (RQ2)

• Participants: Four of the interviewed therapists.
• Method: A 4-hour remote co-design workshop using persona-based design.
• Process:
  1. Persona Design: Therapists created representative DHH client personas (e.g., "Kim," a job seeker with social anxiety; "Choi," a homemaker feeling isolated).
  2. Solution Exploration: Therapists brainstormed how GenAI (Conversational Agents and Music GenAI) could address specific challenges like emotional withdrawal and low creative confidence.
  3. Therapy Process Design: A structured four-stage therapeutic workflow was defined.
• Design Rationale: The workshop established three core design principles:
  • Supportive Empathy: The AI must provide non-judgmental, empathetic responses.
  • Example Response Options: To reduce cognitive load, the AI should offer selectable keywords or phrases for users to choose from.
  • Visual-Based Metaphors: Leveraging the visual strengths of DHH users, the AI should prompt users to describe emotions as scenes, colors, or images rather than abstract auditory concepts.

Phase 3: Usage Study (RQ3)

• Participants: 23 Korean DHH individuals (Cochlear Implant users) who identify as Deaf, are text-literate, and have an interest in music.
• Procedure: Participants engaged in a ~100-minute session using the developed tool under real-time supervision by a music therapist.
• Data Collection: System logs, chat transcripts, pre/post-questionnaires (measuring self-disclosure, satisfaction, and usability), and post-session interviews.

3. System Implementation

The tool is a React-based web application integrating three core modules:

1. Conversational Agent (CA):

   • Backend: Python-based using OpenAI's GPT-4o.
   • Architecture: A state-step prompting framework. The conversation is divided into four states:
     1. Therapeutic Connection: Building rapport and identifying emotional themes.
     2. Making Lyrics: Collaborative writing using visual metaphors.
     3. Making Music: Selecting musical attributes (genre, mood, instruments) based on visual/emotional descriptions.
     4. Song Discussion: Reflective dialogue to interpret the created song.
   • Prompt Engineering: Uses a "General Prompt" (role definition), "State Guidance Prompt" (specific goals per state), and "Required Variable Extraction" (to ensure therapeutic data is collected before moving to the next step).

2. Music Generation:

   • Model: SUNO v3.5 (commercial music GenAI).
   • Input: The CA converts user lyrics and selected attributes (e.g., "sad piano, slow tempo") into text prompts for SUNO.

3. Music Visualization:

   • Function: Converts the generated audio into a visual representation for DHH users.
   • Mapping:
     • Lyrics: Displayed with word-level timing.
     • Pitch: Visualized via vertical text movement.
     • Loudness: Represented by text size.
     • Atmosphere: Indicated by font style and color.
     • Beat: Visualized as synchronized squares.
   • Alignment: Uses the Needleman-Wunsch algorithm to align the AI-generated lyric transcript with the user's edited lyrics for accurate timing.

4. Key Results

The study yielded significant qualitative and quantitative findings regarding the tool's efficacy:

• High Usability and Engagement: All 23 participants completed the session. Average usage time was 40.15 minutes. Usability scores were high, with users reporting low mental load.
• Enhanced Self-Disclosure: Participants reported high levels of self-disclosure (M=6.02/7). The text-based, judgment-free nature of the CA made users feel safer than interacting with humans, allowing them to discuss sensitive topics (divorce, disability, isolation) they usually avoid.
• Effectiveness of CA Strategies:
  • Supportive Empathy: Users felt "heard" and validated, particularly when the AI used proactive empathy (anticipating emotions) rather than just mirroring.
  • Visual Metaphors: Prompts like "What color represents your mood?" successfully helped users translate abstract emotions into concrete lyrics.
  • Example Options: Providing selectable keywords reduced the anxiety of "not knowing what to say."
• Therapeutic Outcomes:
  • Emotional Objectification: Transforming personal struggles into lyrics and music helped users view their problems more objectively.
  • Catharsis: Listening to the co-created music provided emotional release. Even when the AI's output didn't perfectly match the user's intent (e.g., a "calm" request resulting in a "somber" track), users often reframed this mismatch as a new insight into their own feelings.
  • Inner Resource Discovery: Post-song discussion helped users rediscover personal strengths and life purposes.

5. Key Contributions

1. Inclusive AI Design: The study demonstrates how to adapt GenAI for DHH populations by shifting from auditory-centric to multimodal (visual/narrative) therapeutic frameworks.
2. Co-Design Framework: It provides a validated methodology for integrating professional music therapy techniques with GenAI, specifically addressing the unique sensory and emotional needs of DHH users.
3. Empirical Evidence: It offers the first usage study showing that DHH individuals can effectively use GenAI for collaborative songwriting to achieve emotional release, reinterpretation, and deeper self-understanding.
4. Technical Architecture: The paper details a novel pipeline combining LLMs, music generation, and real-time visual synchronization tailored for non-auditory music appreciation.

6. Significance and Future Directions

This research challenges the notion that music therapy is exclusively for hearing individuals. It proposes a paradigm shift where music is treated as a symbolic, multimodal medium for emotional care.

• Clinical Potential: The tool can serve as a "pre-therapy" interface or a between-session homework assignment, helping therapists gain insights into a client's emotional state through generated lyrics and music.
• Self-Help: It functions as a low-threshold self-reflective journaling tool, allowing DHH individuals to process emotions independently.
• Limitations & Future Work: The study was limited to Korean CI users with high text literacy. Future work must involve DHH communities who rely primarily on sign language and test the tool across diverse cultural contexts. Additionally, improving the controllability of music generation models to better match user intent is a critical technical next step.

In conclusion, this study successfully bridges the gap between Generative AI and Music Psychotherapy, creating an accessible, inclusive platform that empowers DHH individuals to heal through the creative act of songwriting.