Driving Plato's Chariot: A Dual-Objective Teacher-Student Prototype for Reason-Affect Allocation in Dialogue

Technical Summary: Driving Plato's Chariot

Problem Statement
Conversational AI systems face a fundamental tension between maintaining factual reliability and responding sensitively to affective context. While affective computing emphasizes recognizing emotional signals and preference-based alignment aims to improve instruction following, human preferences do not guarantee truthfulness (e.g., agreeable responses may be preferred over truthful ones). Existing methods like RLHF or DPO address broad behavioral preferences but often lack explicit mechanisms to coordinate competing objectives like "reason" and "affect" within a single control framework. This paper investigates whether a compact, dual-objective teacher-student architecture can explicitly model and allocate resources between these two competing outputs, while critically auditing the validity of the metrics used to measure success.

Methodology
The study proposes a "Platonic chariot" metaphor, where a shared controller (the charioteer) manages two distinct outputs: a reason-allocation proxy ($r_t$) and an affect-allocation proxy ($e_t$).

• Data and State Representation: The experiment utilizes the first 1,000 conversations (11,936 messages) from a generated dialogue corpus (ychen/Generated-Empathetic-Dialogues-v0.1-Smol). Each message is mapped to a three-dimensional state vector $s_t$:
  1. Length Density: Normalized character count.
  2. Negative-Sentiment Probability: Confidence score from a DistilBERT model fine-tuned on SST-2 (binary sentiment).
  3. Urgency Heuristic: A binary flag triggered by exclamation marks, all-caps, or short length.
• Architecture:
  • Teacher: A dual-headed neural network (3 inputs $\to$ 64 $\to$ 32 $\to$ two independent sigmoid heads) with 2,402 parameters. It outputs the reason and affect allocations.
  • Student: A significantly smaller network (3 inputs $\to$ 32 $\to$ 2 sigmoid outputs) with 194 parameters, trained to mimic the teacher's outputs via Mean Squared Error (MSE).
• Reward and Optimization:
  • The teacher is updated using a scalar reward $R_t$ based on three absolute error terms: deviation of $r_t$ from length density, deviation of $e_t$ from negative sentiment, and a soft penalty if $r_t + e_t \neq 1$.
  • Crucial Limitation: The update rule adds Gaussian noise to the teacher's action and minimizes the MSE between the policy and this noisy action, weighted by the negative reward ($-R_t$). The paper notes this is not a standard policy-gradient estimator; because $R_t$ is always non-positive, the loss function attracts the policy toward all sampled actions, with worse actions receiving larger weights, potentially hindering reward maximization.
  • The student is trained via standard knowledge distillation (MSE) against the teacher's clean outputs.

Key Contributions
The paper positions itself as a "transparent proof-of-concept" and an audit rather than a demonstration of superior AI alignment. Its primary contributions are:

1. Explicit Specification: A fully auditable definition of the dialogue features, dual-headed teacher, scalar reward, and compact student used in the experiment.
2. Quantitative Analysis: A detailed report showing that while a small student can achieve strong in-sample fit to a teacher, the teacher's own reward trajectory deteriorates over time.
3. Construct Validity Audit: A critical separation of measured proxies (length, sentiment probability) from the concepts they are often conflated with (factual truth, empathy). The paper explicitly argues that the current setup does not measure truthfulness or empathy.
4. Redesign Path: Concrete recommendations for future studies, including the use of valid emotion/urgency annotations, proper policy-gradient estimators, and independent evaluation of generated responses.

Results
The experimental results over 25 epochs reveal a divergence between model compression and policy optimization:

• Student-Teacher Fit: The student successfully learned to approximate the teacher's outputs. The training MSE dropped significantly from 0.003040 (Epoch 1) to a minimum of 0.000719 (Epoch 10), ending at 0.001481 (Epoch 25).
• Teacher Reward: Contrary to successful reinforcement learning, the cumulative teacher reward worsened continuously, moving from -23,194.12 to -29,219.67. This indicates the update rule failed to optimize the policy toward higher rewards.
• Factual-Accuracy Proxy: A notebook-defined proxy labeled "factual accuracy" declined from 59.3% to 50.9%. The paper clarifies this metric is merely an alignment with the length-derived target, not a measure of actual truthfulness, as no factual verification was performed.

Significance and Claims
The paper modestly claims to establish implementation and in-sample compression feasibility but explicitly denies establishing improved truthfulness, empathy, or reinforcement-learning convergence.

• The architecture demonstrates that a shared controller can coordinate two scalar outputs and that a much smaller network can replicate this behavior on the training trajectory.
• However, the experiment fails to prove that such a system can optimize for complex human values or factual correctness. The deterioration of the teacher reward suggests the optimization algorithm is flawed for the stated goals.
• The primary significance lies in the corrected interpretation of the measurements: the paper argues that previous claims of "empathic resonance" or "alignment transfer" in similar compact prototypes are unsupported by the data. It calls for a rigorous redesign involving valid targets (e.g., claim verification, validated emotion labels), proper baselines, and human evaluation to move from a "proof-of-concept" to a valid multi-objective dialogue-control study.