Inner Speech as Behavior Guides:
Steerable Imitation of Diverse Behaviors for Human-AI Coordination

1Rakshit S. Trivedi, 2Kartik Sharma, 3David C. Parkes
1Massachusetts Institute
of Technology
2Georgia Institute
of Technology
3Harvard
University
Corresponding author: triver@mit.edu
Spotlight Paper at NeurIPS 2025
Paper Code arXiv
Behaviorist framework

(a) Behaviorist framework: Direct stimulus-response mapping

Cognitive framework

(b) Cognitive framework: Linguistically-mediated action selection

Figure: Contrasting theoretical frameworks for IL. (a) The behaviorist approach models human behavior as a direct mapping from environmental states to actions (st ↦ at), treating cognitive processes as opaque transformations. (b) The cognitive approach instantiated by MIMIC introduces inner speech as a mediational layer (st → mt → at), where mt represents linguistically-structured internal deliberation that enables behavioral diversity and contextual adaptation.

Abstract

Effective human-AI coordination requires artificial agents capable of exhibiting and responding to human-like behaviors while adapting to changing contexts. Imitation learning has emerged as one of the prominent approaches to build such agents by training them to mimic human-demonstrated behaviors. However, current methods struggle to capture the inherent diversity and non-Markovian nature of human behavior and lack the ability to steer behavior at inference time. Drawing inspiration from the theory of human cognitive processes, where inner speech guides action selection before execution, we propose MIMIC (Modeling Inner Motivations for Imitation and Control), a framework that uses language as an internal representation of behavioral intent.

MIMIC employs the novel use of vision-language models as linguistic scaffolding to train a conditional variational autoencoder capable of generating inner speech from observations. A diffusion-based behavior cloning policy then selects actions conditioned on current observations and the generated inner speech. MIMIC enables fine-grained steering of behavior at inference time by conditioning the agent on behavior-specific speech. Experiments across robotic manipulation tasks and human-AI collaboration games demonstrate that MIMIC significantly enhances both behavior diversity and fidelity to human demonstrations while enabling nuanced behavioral steering without training on additional demonstrations.

Method

Standard imitation learning maps states directly to actions, \(s_t \rightarrow a_t\), collapsing the reasoning that produced a demonstration into a single reflex. MIMIC instead inserts a linguistic layer — an inner speech variable \(m\) — between perception and action. Rather than modeling one averaged policy, we treat a human’s behavior as a mixture over intents and recover the intent explicitly:

\[ P_H(a \mid s) = \int p_H(a \mid s, m)\, p_H(m \mid s)\, dm \] Behavior = choosing an intent \(m\) given the situation (\(p_H(m \mid s)\)), then acting on it (\(p_H(a \mid s, m)\)). The same state can yield different actions through different inner speech.
Overview of the MIMIC framework

Figure 2: Overview of MIMIC. A pre-trained vision-language model scaffolds inner speech by discriminating the distinct behaviors present in human demonstrations. We then jointly train (i) a DDPM-T diffusion behavior cloner conditioned on this inner speech and (ii) a VAE-based inner-speech generator conditioned on the recent history of states. At simulation time, inner speech is regenerated periodically from the agent’s own past actions to steer the behavior it produces next.

Why language as the internal representation?

MIMIC follows Vygotsky’s account of inner speech — internalized language that mediates between what we perceive and what we do. Three of its documented properties map directly onto three design choices in the framework:

Predicativity

Inner speech foregrounds relations and actions (“block them from the left”) rather than exhaustively naming objects — it encodes what to do, not what is merely present.

→ VLM-generated behavior descriptions
Semantic Condensation

Rich meaning is compressed into short, abbreviated forms. A verbose description collapses into a compact code that still carries the intent.

→ CVAE latent + annealed KL
Regulatory Dynamics

Inner speech unfolds over time, organizing and re-planning ongoing behavior instead of firing once at the start.

→ Periodic re-generation every \(W\) steps

The MIMIC pipeline

1
Scaffold inner speech from demonstrationsoffline · VLM

We do not have ground-truth inner speech, so we elicit it. Each demonstration \(\tau^{(i)} = (I_1, \ldots, I_T)\) is rendered as a short GIF and shown to a pre-trained vision-language model (GPT‑4o), prompted to “generate the inner thought process that describes the distinctive behavior shown.” This yields a natural-language description \(c^{(i)}\) that discriminates one behavioral style from another. We embed it with CLIP so the label lives in a shared vision-language space:

\[ c^{(i)} = \mathrm{VLM}\!\left(\mathrm{GIF}(\tau^{(i)})\right) \quad\longrightarrow\quad m^{(i)} = \mathrm{CLIP}\!\left(c^{(i)}\right) \] The result is an augmented dataset \(\mathcal{D}_M = \{\,( m^{(i)}, s_t^{(i)}, a_t^{(i)} )\,\}\), pairing every state–action step with its trajectory’s inner-speech embedding.
2
Inner-speech-conditioned behavior clonerdiffusion policy · DDPM-T

The policy is a transformer-based diffusion model (DDPM-T) that generates an action by iteratively denoising, conditioned on both the current state \(s\) and the inner speech \(m\). The state is embedded with a domain-specific encoder (a CNN for pixel observations, an MLP otherwise), inner speech through a small MLP, and the two are fused into the denoising transformer. Training is the standard denoising objective — predict the noise \(\epsilon\) that was added at diffusion step \(k\):

\[ \mathcal{L}_{\mathrm{BC}} = \mathbb{E}_{(s,a,m)\sim \mathcal{D}_M,\, k,\, \epsilon}\left[\, \big\lVert\, \epsilon - \epsilon_\theta(a^{k}, s, m, k)\, \big\rVert^2 \,\right] \] Trained with classifier-free guidance: inner speech is dropped (\(m \rightarrow \varnothing\)) with probability \(p\), so a single policy can act with a specified intent or fall back to unconditioned behavior — and intent can be amplified at inference.
3
Behavior-conditioned inner-speech generatorCVAE

At deployment there is no VLM in the loop, so the agent must produce its own inner speech. A conditional VAE learns to do this from the recent history of behavior \(H_t = \{\, s_{t-W:t},\, a_{t-W:t-1} \,\}\). An encoder \(\Psi_{\mathrm{enc}}\) maps the CLIP-embedded inner speech into a latent \(z\), and a decoder \(\Psi_{\mathrm{dec}}\) reconstructs it from \(z\) and the history:

\[ \mathcal{L}_{\mathrm{IS}} = \big\lVert\, m - \Psi_{\mathrm{dec}}(z, H_t)\, \big\rVert^2 \;+\; \beta\, D_{\mathrm{KL}}\!\left(q_\phi(z \mid m, H_t)\,\big\|\,\mathcal{N}(0, I)\right), \quad z = \Psi_{\mathrm{enc}}(m, H_t) \] This is exactly semantic condensation: the reconstruction term keeps the intent, while the KL term — with \(\beta\) annealed upward during training — squeezes verbose descriptions into a compact latent code, mirroring how elaborate self-talk becomes terse, autonomous inner speech.
4
Inference — generate, regulate, and steerrollout · control

An episode starts with no inner speech, \(m_0 = 0\). Then, realizing regulatory dynamics, the agent refreshes its inner speech every \(W\) steps by sampling a latent from the history-conditioned distribution and decoding it — otherwise it holds the previous thought. Because the update is driven by the agent’s own recent states and actions, intent evolves coherently as the rollout unfolds:

\[ m_t = \begin{cases} \Psi_{\mathrm{dec}}(z, H_t), \;\; z \sim q(\cdot \mid H_t) & \text{if } t \bmod W = 0 \\[6pt] m_{t-1} & \text{otherwise} \end{cases} \]

Steering. Because inner speech is just language embedded in the CLIP space, a user can override it with any target behavior description \(B\) — e.g. “carefully navigate around the other agent to avoid collisions” — and the same trained policy adopts that style, with no additional demonstrations or fine-tuning:

\[ m \leftarrow \mathrm{CLIP}(B) \] Fine-grained, inference-time control of diverse behaviors — the property that distinguishes MIMIC from a policy that only reproduces the average demonstration.

Results

We organize our evaluation around five questions, spanning robotic manipulation (the D3IL benchmark) and human-AI collaboration (Overcooked). Throughout, we compare two inference-time variants of our method — MIMIC-S (success-leaning) and MIMIC-E (diversity/entropy-leaning) — against a strong diffusion behavior-cloning baseline (BC, a DDPM-T policy without inner speech).

Q1

Does MIMIC imitate diverse behaviors with high fidelity on the D3IL benchmark?

Yes. Across Aligning, Sorting, and Stacking (and their vision-based variants), MIMIC matches or exceeds the BC baseline on task success while generating substantially more diverse, human-like behavior (higher behavioral entropy). Crucially, its rollouts stay far closer to the human demonstration distribution — a much lower Wasserstein distance over both visited states and task-completion time — showing the gains are fidelity, not just randomness.

0.61 vs 0.47
Behavioral entropy
Aligning · MIMIC-E vs BC (≈+30% diversity)
0.80 vs 0.66
Success rate
Aligning · MIMIC-S vs BC
≈15×
Closer to human states
Aligning · lower state-Wasserstein

Table 1. MIMIC vs. BC (DDPM-T) on the D3IL benchmark.

Environment Model Success rate ↑ Distance ↓ Entropy ↑ State-wass ↓ Time-wass ↓
Aligning BC 0.66450.11050.47430.696159.034
MIMIC-S 0.80210.06640.41840.045950.569
MIMIC-E 0.72290.08470.61480.049245.397
Aligning-Vision BC 0.18330.18750.0895––
MIMIC-S 0.22290.18850.0849––
MIMIC-E 0.20830.18490.1473––
Sorting-Vision BC 0.7972–0.3596––
MIMIC-S 0.8417–0.3719––
MIMIC-E 0.8083–0.4494––
1 box / 2 box–1 box / 2 box / 3 box
Stacking BC 0.8027 / 0.4879–0.2058 / 0.1503 / 0.10499.43336.51
MIMIC-S 0.8129 / 0.6074–0.1774 / 0.0737 / 0.03940.75345.14
MIMIC-E 0.8213 / 0.5333–0.2115 / 0.1556 / 0.087813.69336.51

↑ higher is better, ↓ lower is better; bold = best per environment. For Stacking, success is reported as 1-box / 2-box and entropy as 1-box / 2-box / 3-box; – marks metrics that are not applicable (Wasserstein distances are infeasible in the vision settings). State-Wasserstein is computed from 5 random rollouts.

Q2

Do MIMIC agents act as effective in-silico human surrogates, coordinating well with human-proxy partners?

Yes. Paired with held-out human-proxy partners in Overcooked, MIMIC agents achieve higher collective reward than BC across every layout — Cramped Room, Coordination Ring, and Asymmetric Advantages — in both state-based and vision-based settings. Because MIMIC reproduces the diversity of human play rather than one averaged strategy, it makes a more realistic partner to evaluate collaborative agents against.

151.8 vs 115.8
Collective reward
Cramped Room · MIMIC vs BC (+36)
108.8 vs 73.6
Collective reward
Cramped Room–Vision (≈+48%)
4 / 4
Layouts improved
every Overcooked setting

Table 2. MIMIC vs. BC (DDPM-T) on the Overcooked environments (collective reward, mean ± std).

Environment Model Collective reward ↑
Cramped room BC115.8 ± 3.86
MIMIC151.8 ± 2.45
Cramped room-Vision BC73.6 ± 6.18
MIMIC108.8 ± 4.84
Coordination ring BC113.0 ± 2.21
MIMIC121.0 ± 1.93
Asymmetric advantages BC215.8 ± 3.04
MIMIC227.6 ± 2.69

Bold = better; higher collective reward is better.

Q3

What does the learned inner speech actually represent?

MIMIC’s premise is that language can encode distinct behavioral intents. To see whether it does, we visualize the CLIP-encoded inner speech (generated with GPT-4o) using t-SNE, across every environment — D3IL (top and in-hand cameras for Aligning, Stacking, Sorting) and Overcooked (blue- and green-hat agents on each layout). The embeddings fall into well-separated clusters, one per behavioral mode: inner speech occupies a structured, behavior-discriminative space. That separability is exactly what makes it a useful conditioning signal — the policy can pick out a mode to imitate, and a designer can point at one to steer toward.

Figure 8 — t-SNE of CLIP-encoded inner speech across all environments
Figure 8. t-SNE visualization of CLIP-encoded inner speech (generated with GPT-4o) for the environments used in our experiments. (a–f) Top and in-hand cameras in the D3IL Aligning, Stacking, and Sorting environments. (g–l) Blue- and green-hat agents in the Overcooked layouts (Cramped Room, Coordination Ring, Asymmetric Advantages). Distinct behaviors form separable clusters in the embedding space.
Q4

Which components of MIMIC drive its performance?

Ablations on the Aligning task isolate each design choice and confirm that MIMIC’s gains come from language-structured inner speech, not merely from extra conditioning:

  • Inner-speech representation. Language-based inner speech beats random vectors and k-means cluster codes on both success and diversity — the linguistic structure is what matters.
  • Embedding model. CLIP’s shared vision-language space outperforms a text-only encoder (MPNET).
  • VLM scaffold. GPT-4o yields the best success; o4-mini produces the most diversity; open-source Qwen2.5-VL is competitive.
  • Regulatory loop. Removing periodic inner-speech regeneration during rollout degrades behavior, validating the every-\(W\)-step update.
Figure 4 — Ablation studies on the Aligning task
Figure 4. Ablation studies on the Aligning task: (a) inner-speech type (language vs. k-means clusters vs. random vectors), (b) embedding model (CLIP vs. MPNET), and (c–d) choice of VLM scaffold (GPT-4o, o4-mini, Qwen2.5-VL).
Q5

Can MIMIC generate desired behaviors on command — i.e., steerable imitation?

Yes. Conditioning the frozen policy on a designer-written behavior description steers it to that style with no additional demonstrations or fine-tuning. Under GPT-4o evaluation, generated rollouts align with the requested behavior (ratings ≈3.8–4.4 / 5) while retaining task success, and the update schedule \((t_0, W)\) trades off success against diversity. Qualitative examples are shown in the Generation & Steering Examples below.

up to 4.4 / 5
Behavior-match rating
GPT-4o eval of steered rollouts
60–74%
Task success
under conditioned behaviors
0
Extra demos needed
training-free steering
Figure 5 — Qualitative steering examples under designer-specified descriptions
Figure 5. Qualitative examples of behaviors conditionally generated under designer-specified descriptions (without periodic updates).

Table 3. GPT-4o evaluation of steerable generation; – denotes no periodic update.

\(t_0\) \(W\) Success rate ↑ Entropy ↑ GPT-4o Eval (1–5) ↑
Top Inhand
Training conditions
––0.60830.46904.03 ± 0.634.05 ± 0.51
10200.74170.38153.98 ± 0.683.98 ± 0.81
49500.60420.51563.88 ± 0.724.05 ± 0.71
Validation conditions
––0.66870.57873.83 ± 0.694.40 ± 0.49
10200.64790.29493.94 ± 0.644.17 ± 0.37
49500.72500.63574.23 ± 0.424.30 ± 0.46

Ratings are GPT-4o’s 1–5 assessment of how well a rollout matches its target description, judged from the Top and Inhand cameras. Bold = best within each condition block.

Generation and Steering Examples

Overcooked

Overcooked example 1

Here, I balance onion placement and soup retrieval to maintain workflow. The agent exhibits strategic placement of onions in the pot while communicating with the other agent to reduce overlap in actions.

Overcooked example 2

The agent adapts its movement to minimize collisions in the cramped space. The agent coordinates movements to minimize delays and optimize task completion.

Overcooked example 3

Carefully navigate around the green-hatted agent to avoid collisions.

D3IL Benchmark

Block stacking example 1

Use a zigzag pattern to correct initial misalignment.

Block stacking example 2

Adopt a systematic approach, sorting one color completely before switching.

Block stacking example 3

I focus on speed over precision, rapidly stacking blocks.

Ask a Question

Have a question about our work? We'd love to hear from you!

We'll respond to your email as soon as possible.

BibTeX

@inproceedings{
trivedi2025inner,
title={Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-{AI} coordination},
author={Rakshit Trivedi and Kartik Sharma and David C. Parkes},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=AwLRF1lZvI}
}