Human-Centered AI in Crisis Counseling: A Three-Tier Governance Framework

In the high-stakes world of crisis counseling, professionals face a unique and exhausting challenge. They must listen to people in the deepest moments of despair, often over the phone or through text, and make life-or-death decisions in a matter of minutes. These counselors are trained to spot subtle signs of danger, offer immediate emotional support, and navigate complex safety plans, all while managing the heavy administrative burden of paperwork and follow-up. The pressure is immense; a missed warning sign can lead to tragedy, while a misdirected response can cause a vulnerable person to disconnect from help entirely. For years, researchers have explored whether artificial intelligence could help by predicting who is at risk of suicide based on language patterns. However, a new line of thinking suggests that simply having a computer predict risk is not enough. The real question is not whether a machine can guess the danger, but how a machine can be safely used to support a human counselor without ever taking the wheel.

A recent paper by Ambika Thangaraju proposes a specific way to answer that question. Rather than imagining a future where computers make the hard calls, the author outlines a three-tier system designed to keep human judgment firmly in charge while using technology to handle the heavy lifting of administration. The framework treats artificial intelligence not as a doctor or a decision-maker, but as a highly organized assistant that works under strict supervision. The core idea is to let the computer manage the flow of information, draft notes, and organize resources, while the human counselor remains the sole authority on assessing risk, deciding on a course of action, and determining if an emergency response is needed. This approach attempts to solve the problem of how to integrate powerful new tools into a field where a single error can have fatal consequences.

The proposed system is built on three distinct layers of responsibility. The first layer focuses on the tasks that happen before, during, and after a call, but strictly excludes any clinical decision-making. In this role, the artificial intelligence acts as a support tool. It might help a counselor by quickly retrieving a safety plan template, drafting a summary of the conversation for later review, or flagging a specific phrase that the counselor has been trained to watch for. Crucially, the system is designed to never assign a risk level to a caller, never diagnose a condition, and never initiate an emergency intervention on its own. If the computer suggests a course of action, a human must review and approve it before it becomes part of the record. This layer also includes safeguards for the caller, such as ensuring they know they are speaking with a system that is listening, and giving them the immediate option to opt out of any automated analysis of their voice or words.

The second layer is where the actual human work happens. This is the domain of the counselor, who retains full authority over the assessment of suicide risk and the final decision on how to help the caller. The framework maps out a clear sequence of steps for these interactions, starting with establishing safety and emotional stability, moving through a structured assessment of the caller's thoughts and plans, and ending with a tailored intervention. Throughout this process, the artificial intelligence from the first layer is available to assist, perhaps by reminding the counselor of a specific checklist or suggesting a resource, but it never overrides the counselor's judgment. If a situation becomes unclear or the counselor feels the information is insufficient to make a safe decision, the system has a specific pathway for that uncertainty. Instead of guessing, the counselor is prompted to seek help from a supervisor. Similarly, if a caller seems to have lost touch with reality, the system recognizes this as a special case that requires immediate, high-level human attention, bypassing standard procedures to ensure safety.

The third layer deals with the organization and the people who run the crisis lines. This tier ensures that the counselors themselves are supported and that the system is working correctly. It includes rules for when a supervisor must review a call, particularly in high-risk situations or when the counselor is unsure what to do. It also addresses the well-being of the staff, recognizing that listening to trauma is emotionally draining. After a particularly difficult call, the system is designed to allow for a period of decompression and a check-in from a supervisor, rather than immediately pushing the counselor to the next call. Furthermore, this layer includes regular audits to make sure the artificial intelligence is behaving as intended and that no automated errors are slipping through. If the system ever needs to make a major change, or if an emergency is declared, the artificial intelligence can be completely locked out, ensuring that only human voices are guiding the crisis response.

The paper emphasizes that this framework is a proposal, not a finished product that has been tested in the field. It is a conceptual design built by combining established methods of crisis intervention with new ideas about how technology should be governed. The author acknowledges that while the models used to build this framework have worked well in face-to-face settings, applying them to anonymous phone and text calls is untested. There are also practical hurdles, such as the cost of the technology, the need for staff training, and the challenge of ensuring the system works fairly across different languages and cultures. The author suggests that the best way forward is to start small, perhaps by using the technology only for drafting notes or organizing resources, and to carefully study how it affects the workload and stress levels of the counselors before expanding its role.

Ultimately, this work offers a cautious and structured vision for the future of crisis care. It argues that the value of artificial intelligence in this field lies not in its ability to replace human intuition, but in its capacity to reduce the administrative burden that often distracts counselors from the people they are trying to help. By creating clear boundaries between what the computer can do and what only a human can decide, the framework aims to protect both the callers and the counselors. It suggests that the most effective use of technology is to serve as a steady, reliable partner that handles the details, allowing the human professional to focus entirely on the complex, empathetic, and critical task of saving a life.