Advances in the Identification and Agreement About Meta-Therapy for Voice Disorders: A Methodological Paper

Background: Meta-therapy (MT) is a powerful dialogue-based element of voice therapy that scaffolds patients’ cognitive models of treatment. MT dialogues have not historically been taught in an explicit manner, and substantial variability in identifying MT exists within and across clinicians. This low reliability is problematic for empirical research and educational transfer. Harnessing contemporary natural language processing technologies to stress test the concept of MT and its clinical instances may enhance our theoretical and empirical grasp of the construct. Method: To capture crucial therapeutic component parts and MT, 10,443 clini cian utterances stemming from conversation training therapy sessions delivered by six expert voice-specialized speech-language pathologists were transcribed and analyzed with a refined annotation framework. Two independent raters anno tated each session and reconciled disagreements through adjudication, and the resulting consensus was compared with an expert gold standard. Time distribu tion was analyzed. Linguistic profiles were derived from bigram frequencies in utterances reaching expert–annotator consensus. Reliability and ambiguity in identification were assessed with multilabel confusion matrices, percentage agreement, Cohen’s kappa, and Gwet’s agreement coefficient (AC1). Results: Gwet’s AC1 indicated substantial-to-almost-perfect agreement for MT in most sessions, outperforming κ and mitigating base-rate artifacts (AC1: up to .98). Mean stand-alone MT duration ranged from 0.32 to 3.88 min per session. Distinc tive MT bigrams contrasted with motor practice or psychosocial collocations that typified direct and counseling content, although lexical overlap produced system atic confusions with MT blended with direct and education/indirect label s. Conclusions: The revised annotation model markedly improved the reproducibility of MT identification and revealed its linguistic signature. The model confirmed MT’s role as a cross-cutting discourse that integrates with other therapeutic modalities. These advances provide an empirical foundation for machine learning classifiers, formal cur riculum content, and further investigation into MT’s contribution to treatment efficacy .