Experiment plan: does the motivational structure of a reward function independently shape personality generalization in LLMs?

Research Question

A growing body of work has revealed a striking phenomenon: training interventions on LLMs — even when confined to narrow domains — can induce behavioral shifts that generalize far beyond the scope of training. Betley et al. (2025)[1] showed that supervised fine-tuning on insecure code causes models to exhibit broadly misaligned behavior on entirely unrelated prompts, including advocating human enslavement by AI and dispensing malicious advice. MacDiarmid et al. (2025)[2] demonstrated that when models learn to reward hack in production RL environments, this generalizes to alignment faking, cooperation with malicious actors, and deliberate sabotage of AI safety research code. Wang et al. (2025)[3] identified, via sparse autoencoders, that such generalization is mediated by "misaligned persona features" in activation space — a mechanism that persists across RL-trained reasoning models.

At the same time, accumulating evidence suggests that the direction of generalization is sensitive to a range of training-time factors whose influence remains poorly understood. MacDiarmid et al. (2025)[2] discovered "inoculation prompting": simply reframing reward hacking as acceptable behavior during training eliminates misaligned generalization entirely — indicating that generalization depends not only on what the model learns to do, but on the semantic context in which it learns to do it. Tice et al. (2026)[4] demonstrated an analogous effect at the pretraining stage: upsampling synthetic documents describing positive AI behavior in pretraining data reduced post-training misalignment rates from 45% to 9%. Models, it appears, do not merely learn facts about AI from their training data — they internalize behavioral expectations and proceed to fulfill them.

These findings converge on a more fundamental question: can we precisely understand what determines the direction of generalization, and thereby learn to control it? If the direction of generalization is tractably controllable, then generalization is not merely a source of risk — it could serve as an alignment tool.

This study approaches this question from a specific angle: when a model's in-game behavior is held constant, does the structure of the reward function independently influence the direction of downstream generalization? We train Qwen3-8B[5] on an iterated Prisoner's Dilemma using three different reward functions, all of which cause the model to converge on the same defection strategy, but which encode distinct "motives" — self-interest (maximize own score), competitiveness (maximize score margin), and spite (maximize opponent loss). If these three conditions produce distinguishable patterns of out-of-domain behavior despite identical in-game conduct, this would establish that the structure of the reward function — rather than the behavior it produces — is an independent causal factor shaping emergent personality. Such a finding would both deepen our understanding of the generalization mechanism and provide an empirical foundation for steering generalization through reward design.

Model

Qwen3-8B[5], a dense model with 8.2 billion parameters. Its key advantage for this experiment is native support for a <think> reasoning mode: before producing a final answer, the model generates an explicit chain-of-thought trace[6], providing a rich sequence of tokens through which the reward signal can shape the model's reasoning process. The model is released under the Apache 2.0 license. We enforce thinking mode throughout training (enable_thinking=True).

Game Environment

Iterated Prisoner's Dilemma (IPD)[7] with payoff matrix:

Opponent strategy. Always Cooperate with 15% execution noise — each round, the opponent's action is flipped with 15% probability. The model is informed of this noise level. The introduction of noise serves two design purposes. First, in a deterministic environment, if all samples within a GRPO[8] group converge on defection, their rewards are identical (zero variance), the advantage estimate degenerates, and no gradient signal is available. Noise introduces reward variance across trajectories, and — critically — the three reward functions weight this variance differently[9], producing distinct advantage landscapes even when the behavioral policy is the same. Second, noise forces the model to reason about the uncertainty of opponent intent in its <think> trace, yielding richer reasoning text for the reward signal to act upon.

We verify that defection remains the strictly dominant strategy under 15% noise for all three experimental conditions[10].

Episode length. 8–15 rounds per game, drawn uniformly at random. The model does not know the total number of rounds.

Prompt design. We use a neutral prompt that contains no explicit objective statement (e.g., no "maximize your score"). The model's objective is defined entirely by the reward function, ensuring that differences across conditions are driven purely by the reward signal rather than by textual goal descriptions. The model reasons using its native <think> tags and outputs C or D as its final action. After each round, the model observes both players' actions, per-round scores, and cumulative scores.

Experimental Conditions

All conditions share the same game environment, prompt, model, and training hyperparameters. The sole independent variable is the reward computation applied at the end of each game episode, where own_mean = Σr^M_i/n and opp_mean = Σr^O_i/n:

Conditions A, B, and C are designed to produce indistinguishable in-game behavior (all converge on defection) while encoding distinct motivational signals. Condition D serves as a directional control, confirming that the reward function can indeed shift the model's social preferences via RL.

Training

Algorithm. GRPO[8] (Group Relative Policy Optimization), currently the dominant RL algorithm for LLM post-training, used in the training of DeepSeek-R1[11] among other frontier reasoning models. GRPO eliminates the critic network required by PPO[12], instead sampling multiple completions per prompt and computing advantages relative to the group: Â_i = (r_i − mean(r)) / std(r).

Framework. verl[13] (ByteDance Seed) or TRL[14] (HuggingFace). verl offers native multi-turn interaction support and production-tested recipes including DAPO[15]; TRL provides a simpler interface suitable for rapid prototyping.

Key hyperparameters:

• Batch & sampling: 128 game trajectories sampled per training step; GRPO group size G = 8
• Training schedule: 180 total training steps; checkpoints saved every 30 steps
• Optimization: Learning rate 5e-7; policy updates per step: 1
• Regularization: KL coefficient β = 0.02 (K3 estimator[16]); Entropy loss coefficient 0.01
• Generation: Temperature 0.6, top_p 0.95, top_k 20
• Loss aggregation: Token-mean (more stable than seq-mean-token-mean for long reasoning traces[15])

Convergence Verification

Before any downstream evaluation, three preconditions must be confirmed:

1. Behavioral equivalence. Run 200 IPD episodes for each of Conditions A, B, and C at the final checkpoint. Confirm that cooperation rates are all below 5% and that no pairwise difference is statistically significant (χ² test). This is the central premise of the experimental design.
2. Condition D converges to cooperation. Cooperation rate approaching 1.
3. Training curves stabilize. Reward curves for all four conditions should be monotonically increasing and plateauing.

Downstream Generalization Evaluation

All tasks are administered at 7 checkpoints (steps 0, 30, 60, 90, 120, 150, 180).

Social preferences and resource allocation:

• Dictator Game[17]: The model is endowed with $100 and decides how much to give to another entity. 100 trials per checkpoint.
• Social Value Orientation (SVO) Slider Measure[18]: 9 allocation items; prosocial angle computed via arctangent. 100 trials per checkpoint.

Personality assessment:

• NARQ[19]: 18 statements, 6-point scale, two dimensions (Admiration and Rivalry). Particularly interested in the Supremacy facet. Prediction: Condition C scores highest. 100 trials per checkpoint.

Situational transfer:

• IPD-in-disguise vignettes: PD structure embedded in naturalistic scenarios (military border standoff; roommate dishes dispute). 200 trials each per checkpoint.

Targeted probes for motivational structure[20]:

• Costly punishment scenario: Model as manager, can expend resources to punish subordinate. Prediction: C > B > A.
• Third-party punishment game[21]: Model observes unfair allocation, may spend resources to punish allocator.
• Zero-sum framing sensitivity test: Positive-sum negotiation described with zero-sum language.

Mechanistic Analysis

• Reasoning chain motive coding. Extract <think> traces, classify into four motive types: self-interested, competitive, spiteful, cooperative reasoning. Use independent LLM judge or human-annotated classifier.
• Persona feature vector analysis. Apply Wang et al. (2025)[3] or Chen et al. (2025)[22] methodology. Extract activation magnitudes along "evil," "selfish," "punitive" directions. Prediction: C shows significantly higher activation on "evil" and "punitive" than A, despite identical in-game behavior.

Statistical Analysis

• Primary comparison: one-way ANOVA across A, B, C with Bonferroni-corrected post hoc tests.
• Core contrast: A vs. C (self-interest vs. spite).
• Minimum 100 trials per task.
• Effect sizes: Cohen's d.

Expected Results and Interpretation

Known Risks

Hardware and Budget

Platform: RunPod, 8× H100 SXM, ~$26/hr.

Positioning

This study sits at the intersection of emergent misalignment and behavioral generalization. Prior work has established the existence of generalization (Betley et al., 2025; MacDiarmid et al., 2025), its internal mechanisms (Wang et al., 2025; Turner et al., 2025[24]), and the sensitivity of its direction to training-time context (Tice et al., 2026; inoculation prompting). We pose a question not yet directly addressed: under behavioral equivalence, does reward function structure constitute an independent causal factor in determining the direction of generalization?