💻 computer science

Modeling Engagement with Brand and Organizational TikTok Videos Using Machine-Assisted Theory-Ensemble Annotation

This paper demonstrates how multimodal large language models can overcome the scalability challenges of manually annotating complex audiovisual and theoretical variables in short-form video, enabling the analysis of 10,000 TikTok videos to identify which narrative and structural factors drive engagement and provide actionable insights for content creators.

Original authors: Sander Paekivi, Andres Karjus

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sander Paekivi, Andres Karjus

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand why some short videos on TikTok go viral while others get ignored. Usually, researchers just look at easy-to-count things: "How many followers does the account have?" or "How old is the video?" But this is like judging a book only by its cover size and publication date, ignoring the story inside.

This paper tries to read the "story" inside thousands of TikTok videos made by brands and organizations in Estonia. Here is a simple breakdown of what they did and what they found.

The Problem: Too Much Video, Too Little Time

Short videos are a mix of music, talking, text, editing, and trends. To understand them, you usually need a human to watch and write down notes about the "vibe," the story structure, or the emotional tone. But doing this for 10,000 videos by hand is impossible—it would take years.

The Solution: The "Super-Reader" Robot

The researchers used a powerful AI (a Large Language Model) to act as a "super-reader." They taught this AI to watch the videos and fill out a very specific, complex checklist based on old-school theories about storytelling, advertising, and psychology.

Think of the AI as a detective with a magnifying glass that can instantly spot 77 different clues in a video, such as:

Is the speaker talking directly to you?
Is the music happy or sad?
Is there a clear "hero" and a "villain"?
Is the video trying to sell something or just make you laugh?

The Experiment: The "Human vs. Robot" Check

Before trusting the robot, the researchers had three human students watch a small sample of videos and fill out the same checklist.

The Good News: The robot was very good at spotting obvious things, like "Is there music?" or "Is the camera moving fast?" Humans and robots agreed on these easily.
The Bad News: When it came to deep, abstract ideas (like "What is the hidden symbolic meaning?" or "What is the philosophical conflict?"), even the humans couldn't agree with each other. The robot struggled here too. It's like asking a group of people to agree on the "true meaning" of a dream; everyone sees something different.

The Findings: What Actually Gets Likes?

The researchers then used the robot's notes to predict which videos would get the most Likes and Comments. They compared the robot's detailed notes against the simple "follower count" baseline.

1. The "Packaging" Matters More Than the "Deep Meaning"
The robot found that the most successful videos weren't necessarily the ones with the deepest philosophical stories. Instead, success was driven by how the video was packaged:

Audio: Videos with original spoken voices (people talking) got more likes than those with just music.
Intent: Videos meant to be funny or entertaining performed better than those meant to be purely educational.
Setting: Videos filmed in recognizable indoor commercial settings (like a shop or office) did well.

2. The "Call to Action" is the Key to Comments
If you want people to comment, the video needs to ask for it. Videos that used "engagement bait" (directly asking questions or prompting a reaction) got significantly more comments. Practical "how-to" videos also sparked more questions and information-seeking comments.

3. The "Familiarity" Factor
The study found a surprising result: Being similar to the average works.

Videos that looked and felt like the "standard" successful videos in this group (using common audio, common humor, common settings) tended to get more likes than videos that tried to be weirdly unique.
It's like a restaurant: If you want a steady stream of customers, it's often safer to serve the popular dish everyone expects than to serve a weird, experimental meal that confuses people.

The Bottom Line

This paper shows that we can use AI to turn "vibes" and "stories" into hard data. While the AI can't perfectly understand deep human philosophy, it is very good at spotting the structural ingredients of a video.

For a brand making a TikTok, the takeaway isn't "write a deep, complex story." The takeaway is: Use clear audio, keep it entertaining, set it in a familiar place, and ask people to comment. The "magic" isn't in the hidden meaning; it's in the visible, repeatable patterns that the AI helped identify.

Technical Summary: Modeling Engagement with Brand and Organizational TikTok Videos Using Machine-Assisted Theory-Ensemble Annotation

Problem Statement
Short-form video content on platforms like TikTok presents significant challenges for large-scale research. Meaning in these videos emerges from complex, multimodal interactions between audiovisual elements, language, editing, and algorithmic dynamics. Traditional manual annotation of these layers is labor-intensive, difficult to standardize, and often limited to small samples, hindering the ability to study cultural production and engagement at scale. Existing large-scale studies frequently rely on easily measurable predictors (e.g., account size, timing, network structure) or general multimodal features, often overlooking deeper interpretive questions regarding narrative structure, rhetorical address, and semiotic organization. While multimodal Large Language Models (LLMs) show promise, there is a need for a rigorous workflow that operationalizes long-standing interpretive theories into structured variables without losing theoretical nuance or interpretability.

Methodology
The study employs a machine-assisted "quantitizing" approach, converting qualitative theoretical judgments into quantifiable categorical variables.

Corpus Construction: The dataset comprises 9,654 videos from 30 curated brand and organizational accounts in Estonia (a linguistically bounded ecosystem of 1.3 million people). The videos span from July 2020 to January 2026, with the majority posted between 2023 and 2025.
Annotation Design: The researchers utilized a single video-capable frontier model (Gemini 2.5 Pro) to annotate the corpus. The annotation scheme is "theory-saturated," deriving 77 structural variables from 11 distinct theoretical blocks, including:
- Narratology (Labovian structure, Propp/Campbell arcs, Polti conflict types).
- Rhetoric (Aristotelian appeals, Burke's dramatistic pentad).
- Semiotics (Chandler's sign modes, Hall's encoding/decoding).
- Multimodal frameworks (Grzenkowicz & Wildfeuer).
- Communication theories (Uses and gratifications).
  The model was prompted to output a structured JSON object containing binary/categorical variables and diagnostic text fields, avoiding free-form prose to ensure consistency.
Validation: A subset of 100 videos was manually annotated by three human coders to assess reliability. Agreement metrics (Krippendorff's $\alpha$ , Fleiss' $\kappa$ ) and LLM accuracy against human majority votes were calculated. Results indicated a reliability gradient: perceptual and communicative variables (e.g., audio type, camera angle) showed high agreement, while deeper semiotic and archetypal constructs (e.g., Burke's pentad, Polti conflicts) showed low agreement, often due to human annotator difficulty rather than solely model failure.
Quantitative Analysis:
- Predictive Modeling: Grouped CatBoost models were trained to predict engagement metrics (likes, comments, shares, saves) and comment section activity (supportive vs. information-seeking). The models enforced "grouped out-of-account" splits to prevent data leakage, ensuring videos from the same account did not appear in both training and test sets. Baseline controls included account follower count and video age.
- Interpretability: A secondary linear mixed-effects model analyzed the top predictors from the CatBoost models. Additionally, a "design choice" analysis simulated marginal effects by varying single variables while holding others constant to identify high-leverage structural features.
- Similarity Mapping: Videos were mapped in a 2D configuration space using Hamming distance over the full annotation set to explore structural similarity and its relationship to performance.

Key Contributions

Operationalization of Theory: The study demonstrates a workflow for translating complex, long-standing interpretive theories (narratology, semiotics, rhetoric) into a structured, machine-readable annotation scheme for short-form video.
Machine-Assisted Quantitizing: It validates the use of multimodal LLMs to annotate large batteries of theoretically motivated variables, showing that while deep semiotic constructs remain challenging, perceptual and communicative variables can be coded with sufficient reliability for exploratory analytics.
Empirical Baseline: It provides a comprehensive dataset of Estonian brand and organizational TikTok content, offering a delimited platform repertoire that avoids the noise of isolated viral cases.
Methodological Framework: The paper establishes a reproducible pipeline involving explicit codebooks, validation samples, and bias checks to transform qualitative distinctions into structured variables.

Results

Predictive Gains: The theory-coded content variables provided modest but consistent gains over baseline controls (account size and video age). The full models achieved $R^2$ values of 0.27 for likes and 0.18 for comments, representing an incremental improvement of 0.056 to 0.078 over control-only models. Likes and saves were the most predictable outcomes; shares remained the most difficult to predict.
Key Predictors:
- Likes: Most strongly associated with broad packaging choices: original spoken audio (highest contrast), entertaining intent, and indoor-commercial settings.
- Comments: "Call-to-action" (CTA) type was the dominant predictor, specifically "engagement bait." Information-seeking comments were linked to tutorial formats and logos-led rhetoric, while supportive comments correlated with pathos-led rhetoric and travel/fashion topics.
Reliability Gradient: Human validation confirmed that perceptual variables (audio, visual, editing) are coded reliably by both humans and LLMs. In contrast, deeper interpretive variables (Burke, Polti, Chandler) showed poor reliability, with LLMs often over-detecting conflict or semiotic complexity where humans saw straightforward representation.
Structural Similarity: Videos that aligned with the "modal profile" (most common values) of the corpus or the specific account tended to perform better in terms of likes. However, this was not a uniform effect; specific deviations (e.g., breaking the corpus norm for promotional intent) could also be favorable. The advantage of similarity was driven by high-leverage packaging variables rather than total conformity.

Significance and Claims
The paper claims that this approach supports three primary aims:

Exploratory Cultural Analytics: Mapping variation in short-form video culture by making large batteries of theoretical variables observable.
Predictive Modeling: Improving the forecasting of platform dynamics and audience feedback beyond simple visibility metrics.
Diagnostics for Creators: Providing data-driven guidance for content creators to choose between structural and narrative strategies, identifying which subsets of variables carry signal for specific niches.

The authors emphasize modesty in their claims: the study is associational, not causal. The value of the LLMs lies not in perfect accuracy for every theoretical construct, but in making the assessment of large, diverse theoretical frameworks feasible. This allows researchers to identify which specific variables carry predictive signal and which are merely descriptive, translating theoretical insights into actionable guidance for content strategy within a specific niche. The findings suggest that while deep semiotic analysis remains difficult to standardize, the operationalization of narrative and rhetorical structures offers a viable path for understanding the "why" behind video engagement.