💬 NLP

Comparing How a Chatbot References User Utterances from Previous Chatting Sessions: An Investigation of Users' Privacy Concerns and Perceptions

This study investigates how different chatbot referencing strategies (none, verbatim, or paraphrase) affect user engagement and privacy concerns, finding that while verbatim and paraphrase methods increase perceived intelligence and engagement, verbatim referencing specifically triggers higher privacy worries.

Original authors: Samuel Rhys Cox, Yi-Chieh Lee, Wei Tsang Ooi

Published 2026-02-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Samuel Rhys Cox, Yi-Chieh Lee, Wei Tsang Ooi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are having a conversation with a digital friend, a chatbot. Every week, you tell it about your daily habits—specifically, how often you use dental floss. Now, imagine this digital friend has a memory. The big question the researchers asked is: How should this friend bring up things you said last week?

Should it be like a robot reading from a script, a human summarizing the gist, or should it just pretend it doesn't remember at all?

The researchers at the National University of Singapore set up a three-week experiment to find out. They treated the chatbot like a "memory machine" with three different settings:

The "Silent" Mode (None): The chatbot asks new questions but never mentions what you said last time. It's like talking to someone who has amnesia every time you meet.
The "Photocopier" Mode (Verbatim): The chatbot quotes you word-for-word. If you said, "My teeth hurt when I floss," it replies, "Last week you said, 'My teeth hurt when I floss'." It's like a parrot repeating your exact words.
The "Human" Mode (Paraphrase): The chatbot remembers the idea but rephrases it. If you said, "My teeth hurt," it replies, "Last week you mentioned your teeth were hurting." It's like a friend saying, "Oh yeah, you told me about that sore tooth."

The Good News: Smarter and More Engaging

When the researchers asked the participants how they felt, the "Silent" mode was the clear loser. People thought the chatbot was boring and didn't seem to be listening.

However, both the Photocopier and the Human modes made the chatbot feel much smarter and more engaging. It was as if the chatbot had suddenly woken up and started paying attention. People felt more connected because the bot remembered their story.

The Bad News: The "Creepy" Factor

Here is where things get tricky. While the Photocopier mode made the bot seem smart, it also made some people feel a little uneasy.

When the bot repeated your words exactly, some participants felt a bit like they were being recorded by a surveillance camera. One person described it as "unnerving." It felt like the bot was holding up a mirror too perfectly, making people realize, "Wait, it's actually saving everything I say."

The Human mode (paraphrasing) was a sweet spot for many. It felt natural, like a real conversation, without triggering that "big brother is watching" feeling as strongly as the exact quotes did.

The "Privacy Paradox"

The study highlights a classic tug-of-war called the Personalization-Privacy Paradox.

On one side: You want the bot to remember you so it feels like a real, intelligent friend.
On the other side: You don't want it to remember too well, because that feels like an invasion of your private space.

The researchers found that while people wanted the bot to remember them, the way it did so mattered. If it was too literal (verbatim), it felt invasive. If it was too vague (none), it felt useless.

What Did the Interviews Reveal?

The researchers also sat down with 15 people to ask, "Why did you feel that way?"

The "Tool" vs. The "Friend": Some people saw the chatbot as a tool, like a calculator. They preferred the Photocopier mode because they wanted their exact words saved for accuracy, like a contract. Others saw the bot as a Friend, and they preferred the Human mode because it felt warmer and less robotic.
The "Guilt Trip": For some, hearing their exact words repeated back (especially about not flossing enough) felt like a guilt trip or a "creepy" reminder of their failures.
The "Lost in Translation" Fear: People using the "Silent" mode worried the bot was too dumb to understand them, so they kept their answers simple. People using the "Human" mode felt the bot was listening, but they worried the bot might twist their meaning.

The Bottom Line

If you are designing a chatbot that talks to people over time, you have to walk a tightrope.

Don't be a ghost: Ignoring past conversations makes the bot seem stupid and uninterested.
Don't be a parrot: Repeating words exactly can feel creepy and invasive.
Be a good listener: Summarizing what the user said (paraphrasing) usually strikes the best balance. It feels smart and friendly without making people feel like they are being spied on.

The study concludes that the best approach depends on what the user wants: if they view the bot as a tool for accuracy, exact quotes might be okay. But if they view it as a conversational partner, rephrasing the memory is the way to go.

Technical Summary: Comparing How a Chatbot References User Utterances from Previous Chatting Sessions

Problem Statement

Advances in language models enable chatbots to maintain long-term memory and reference previous user utterances across multiple sessions. While this capability can enhance social presence, build rapport, and improve user engagement in domains like open-domain conversation and personal health, it introduces a tension known as the Personalization Privacy Paradox. This paradox describes the conflict between the benefits of personalized services (which require data collection) and user feelings of intrusiveness or privacy violation.

A specific gap exists in understanding how the format of these references impacts user perception. Designers must choose between referencing styles ranging from low social presence (no explicit reference) to high social presence (verbatim quotes or paraphrases). It remains unclear whether verbatim referencing (directly quoting user input) or paraphrasing (summarizing user input) creates a better balance between perceived intelligence/engagement and privacy concerns.

Methodology

The authors conducted a three-week longitudinal between-subjects study ( $N=169$ ) followed by semi-structured interviews ( $N=15$ ).

Experimental Design

Task: Participants interacted with a chatbot once a week for three weeks regarding their dental flossing habits (a domain chosen for its personal nature but varying sensitivity).
Independent Variable: Chatbot Referencing Format with three levels:
1. None (Control): The chatbot did not explicitly reference previous user utterances, only referencing discussions at a high level.
2. Verbatim: The chatbot incorporated previous user utterances exactly as spoken (e.g., "Last week you said 'My teeth sometimes hurt when I floss'").
3. Paraphrase: The chatbot incorporated paraphrased versions of previous utterances (e.g., "Last week you said that your teeth hurt").
Implementation: The chatbot was hosted on Qualtrics using JavaScript/HTML. Microsoft LUIS was utilized for intent recognition and to select pre-written paraphrases corresponding to user intents.
Measures:
- Weekly: Ratings on 7-point Likert scales for desire to continue usage, perceived engagement, and perceived intelligence/competence.
- Final Week (Week 3): Specific measures for privacy concerns (data collection fears), privacy intrusiveness (feeling of encroachment), and privacy risks (uncertainty of adverse consequences).

Qualitative Component

Following the quantitative study, 15 participants (5 per condition) underwent semi-structured interviews. These sessions explored user reasoning regarding privacy, intelligence, and preference, culminating in a "think-aloud" ranking of the three referencing formats after participants were informed of all conditions.

Key Results

Quantitative Findings

Positive Perceptions (Intelligence & Engagement):
- Both Verbatim and Paraphrase conditions were rated significantly higher than the None condition regarding perceived chatbot engagement and intelligence.
- Explicitly referencing past utterances made the chatbot appear more competent and attentive.
- There was no significant difference in the desire to continue using the chatbot across the three conditions.
Privacy Perceptions:
- Verbatim referencing generated significantly higher privacy concerns than the None condition regarding the specific measure: "I was concerned that the chatbot was collecting too much personal information about me" ( $p=0.0227$ ).
- Intrusiveness: Verbatim and Paraphrase conditions showed weakly significant trends toward being perceived as more intrusive than None, though these did not reach strict statistical significance ( $p < 0.10$ ).
- Overall Privacy Levels: Despite the differences, average privacy concern scores across all groups remained below the neutral midpoint ("Neither Agree Nor Disagree"), suggesting that for the dental flossing domain, privacy violations were not perceived as severe.

Qualitative Findings (Interviews)

Privacy Concerns:
- Verbatim: Some users found direct quoting "unnerving," "creepy," or "scary," as it made data storage and manipulation highly apparent. However, a subset of users preferred Verbatim because they distrusted the chatbot's ability to paraphrase accurately or felt that direct quotes were more transparent (showing data was stored but not altered).
- Paraphrase: Generally viewed as more natural and human-like. However, some users worried that paraphrasing involved "processing" their data, which could be invasive.
- None: Users felt the chatbot was disengaged or unintelligent. Some participants simplified their responses, doubting the bot could understand complex sentences without explicit feedback.
Intelligence & Naturalness:
- Verbatim and Paraphrase were both seen as signs of intelligence.
- Paraphrase was often described as feeling like a "friend" remembering a conversation, whereas Verbatim felt like a "survey" or "copy-paste."
User Preference:
- In the final ranking, None was universally the last choice.
- Paraphrase was the first choice for 10/15 interviewees (viewing the bot as a conversational partner).
- Verbatim was the first choice for 5/15 interviewees (viewing the bot as a tool and valuing the fidelity of their own words).

Key Contributions

Empirical Evidence on Referencing Formats: The study provides data comparing three distinct referencing strategies (None, Verbatim, Paraphrase) in a longitudinal setting, moving beyond hypothetical scenarios.
Validation of the Trade-off: It confirms the Personalization Privacy Paradox in the context of chatbot memory: while referencing (both Verbatim and Paraphrase) increases perceived intelligence and engagement, Verbatim referencing specifically heightens concerns about the quantity of data collection.
Nuanced User Preferences: The research identifies that user preference is not monolithic; it depends on whether the user views the chatbot as a "tool" (preferring Verbatim for accuracy) or a "conversational partner" (preferring Paraphrase for naturalness).
Design Implications: The paper suggests that a "one-size-fits-all" approach is insufficient. Designers may need to mix formats (e.g., paraphrasing behavior but quoting specific goals) or tailor formats based on user personality and the sensitivity of the domain.

Significance and Claims

The paper claims that the format of referencing user utterances is a critical design lever that directly influences user trust and engagement. The authors assert that:

Explicit referencing is generally beneficial for engagement and perceived intelligence compared to non-referencing.
Verbatim referencing carries a specific privacy cost, making users more conscious of data collection, though this may be mitigated by the low sensitivity of the specific topic (dental hygiene) used in the study.
Context matters: The appropriateness of a format depends on the user's goal (e.g., accountability vs. conversation) and their perception of the chatbot's role.

The authors maintain a modest stance, noting that their findings are specific to the dental flossing domain and a three-week timeframe. They do not claim that Verbatim is universally "bad" or Paraphrase universally "good," but rather that the choice must be informed by the specific context, the sensitivity of the data, and the user's mental model of the chatbot. The study concludes by recommending that designers consider transparency and user control when implementing these referencing features.