Development and validation of an approach to automated qualitative content analysis using locally deployed Large Language Models (LLMs).
Problem Statement: Qualitative content analysis enables the systematic analysis of large volumes of qualitative data and offers considerable potential for both qualitative and quantitative analysis (Galle & Iten, 2027). Conversion mixed-methods designs (Tashakkori et al., 2021) allow qualitative data to be examined from different analytical perspectives. However, coding large volumes of textual data is resource-intensive. This is particularly challenging when sufficiently large samples are required for quantitative analyses. Large Language Models (LLMs) offer new opportunities to automate these analytical processes. Nevertheless, it remains unclear to what extent LLMs can reliably meet the methodological requirements of qualitative content analysis, particularly regarding segmentation, category development, and coding.
Research Aims and Questions: The AUQUIL project aims to develop and validate a locally deployed application that autonomously analyses textual data in accordance with structuring qualitative content analysis (Kuckartz & Rädiker, 2022, 2024). Specifically, the application is designed to develop a deductive category system, segment textual data into analytically meaningful units, assign these units to categories, and inductively refine the category system.
The project addresses the following overarching research question: Under what conditions can locally deployed LLMs automate key stages of structuring qualitative content analysis (Kuckartz & Rädiker, 2022, 2024) while adhering to its methodological principles?
Three areas of investigation are derived from this question:
Segmentation: How reliably do LLMs identify analytically meaningful units in textual data?
Category Development: To what extent can LLMs develop and revise categories to produce a coherent category system with clearly distinguishable categories?
Coding: To what extent do automated coding decisions agree with human coding decisions, and how do different language models and review procedures affect analytical quality?
Methodological Approach: The development follows the principles of content-structuring qualitative content analysis with deductive–inductive category development, as proposed by Kuckartz and Rädiker (2022, 2024). Methodological validation focuses on assessing the quality of segmentation, category development, and coding using qualitative interview data. Automated analytical results are compared with analyses conducted by human researchers. In addition, different language models and multi-stage review procedures are evaluated in terms of reliability, consistency, and efficiency. The analytical system is intended to support both qualitative analysis and the conversion of qualitative coding into quantitative data for conversion mixed-methods analyses (Tashakkori et al., 2021).
Current Development and Outlook: A functional prototype has already been developed using Python, Ollama, and locally deployed LLMs. Initial development and testing with qualitative interview data aim to identify methodological and technical challenges and develop appropriate solutions. Systematic validation across different datasets and category systems has yet to be conducted. AUQUIL is particularly intended for research and evaluation projects seeking to analyse large volumes of qualitative data within a conversion mixed-methods design (Tashakkori et al., 2021), but with limited time and financial resources. Running the language models locally allows data to be processed without transferring the material to external model providers. AUQUIL thus aims to contribute to methodologically rigorous and transparent qualitative data analysis using LLMs.
Literatur
Galle, M., & Iten, G. (2027). Kodierungen analysieren und visualisieren. Mixed-Methods-Datenanalyse mit MAXQDA und R. Springer.
Kuckartz, U., & Rädiker, S. (2022). Qualitative Content Analysis: Methods, Practice and Software (2nd ed.). Sage.
Kuckartz, U., & Rädiker, S. (2024). Qualitative Inhaltsanalyse. Methoden, Praxis, Umsetzung mit Software und künstlicher Intelligenz (6., überarbeitete und erweiterte Auflage). Beltz.
Tashakkori, A., Johnson, R. B., & Teddlie, C. (2021). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences (2nd ed.). SAGE.