Enhancing Human-Likeness in Reinforcement Learning Agents via Hierarchical Macro Action Quantization

† indicates joint first author.

Retrocausal, Inc.

Abstract

Human-like agents are a long-standing goal of artificial intelligence. Despite strong performance, most reinforcement learning (RL) agents remain reward-driven and often exhibit behaviors that differ from humans, limiting interpretability and reliability. In this work, we introduce a novel human-like RL framework that predicts action sequences closely aligned with human behaviors while maximizing rewards. Specifically, we encode human demonstrations into macro actions using a hierarchical macro action quantization approach (termed HiMAQ) consisting of two successive levels of vector quantization. The lower quantization level maps input actions to fine-grained subaction clusters, while the higher quantization level aggregates these subaction clusters into action clusters. Extensive evaluations on the D4RL benchmarks show that our hierarchical approach outperforms the non-hierarchical baseline (MAQ), achieving better human-likeness scores while maintaining comparable or better success rates than previous RL agents. The improvements generalize across integrations with various RL algorithms, namely IQL, SAC, and RLPD.

Motivation

As robots take on increasingly complex tasks alongside humans, a key expectation emerges: motion that is not merely successful, but recognizably human-like. Conventional reward-driven RL policies routinely produce unnatural, jerky motions that erode trust and fail to transfer the rich knowledge encoded in human demonstrations.

Recent work has begun to close this gap by grounding RL within the space of human demonstrations. Human-like receding horizon control (HRC) constrains trajectory search to human-generated segments, producing more structured motion. MAQ extends this by encoding human trajectory segments into a discrete codebook via a conditional VQ-VAE, allowing a policy to select from a compact set of learned macro actions. However, MAQ’s single-level codebook treats each macro action as a single atomic prototype, limiting its ability to capture the full variation within each motion segment. A reaching motion, for instance, spans a broad approach, fine grasp alignment, and controlled contact — collapsing all of this into one prototype loses the subtle dynamics that make motion feel human.

We propose HiMAQ (Hierarchical Macro Action Quantization), a two-level vector quantization framework that captures human demonstrations at both the macro action level and the fine-grained subaction level. The key insight is that instead of quantizing an encoded trajectory directly to an action prototype, HiMAQ first maps it to a fine-grained subaction prototype, which is then mapped to a coarser action prototype, preserving both coarse structure and fine-grained detail.

Approach

HiMAQ represents macro actions from offline human demonstrations using a two-level hierarchical codebook structure: a subaction codebook Z = {zj} with αK entries, and an action codebook A = {ai} with K entries.

Stage 1 — Human Demonstration Encoding. The encoder takes the current state st and an H-step macro action mt as input, producing a latent embedding et. The first level of quantization maps et to the nearest subaction prototype qZt in codebook Z via Euclidean distance. The second level then maps qZt to the nearest action prototype qAt in codebook A. Finally, qAt and st are passed to a decoder that reconstructs the macro action m̃t. Training uses a combination of reconstruction loss, commitment losses, and codebook losses across both quantization levels.

Stage 2 — Reinforcement Learning. A policy πθ is trained to select an index from the subaction codebook Z, producing qZt. The corresponding action prototype qAt is then retrieved from A, and the decoder reconstructs the macro action for environment interaction. By operating in the discrete subaction space, the policy stays within the human demonstration manifold while retaining the fine-grained variation that single-level quantization discards.

teaser

HiMAQ Framework. (a) Human demonstration encoding via HiMAQ. The state (st) and macro action (mt) are embedded by an encoder into latent vector et. Next, et is mapped to the nearest subaction prototype qZt, which is then mapped to the closest action prototype qAt. The decoder yields the predicted macro action m̃t. (b) Reinforcement learning with HiMAQ. The online policy πθ takes st as input and predicts the subaction prototype qZt, from which qAt and the macro action are derived.

Experimental Results

We evaluate HiMAQ on four dexterous manipulation tasks from D4RL’s Adroit benchmark — Door, Hammer, Pen, and Relocate — each controlled by a 24-DOF simulated hand with 25 human demonstration trajectories. We compare against BC, vanilla RL baselines (IQL, SAC, RLPD), and their MAQ-augmented counterparts. Human-likeness is measured by Dynamic Time Warping (DTW) and Wasserstein Distance (WD) on both states and actions; higher normalized scores indicate more human-like behavior.
 
Results with IQL Baseline
HiMAQ+IQL improves average DTWs from 0.63 to 0.66 and success rate from 0.39 to 0.64, with a particularly large gain on the Hammer task (success rate from 0.00 to 0.87).
Task Metric BC IQL MAQ+IQL HiMAQ+IQL
Door DTWs (↑) 0.18 0.43 0.84 0.87
DTWa (↑) 0.42 0.61 0.95 0.96
WDs (↑) 0.32 0.48 0.75 0.77
WDa (↑) 0.41 0.50 0.81 0.82
Success (↑) 0.02 0.16 0.93 0.95
Hammer DTWs (↑) -0.16 -0.14 0.64 0.68
DTWa (↑) 0.47 0.45 0.92 0.94
WDs (↑) 0.11 0.12 0.75 0.73
WDa (↑) 0.30 0.30 0.84 0.87
Success (↑) 0.00 0.01 0.00 0.87
Pen DTWs (↑) 0.53 0.34 0.55 0.57
DTWa (↑) 0.58 0.51 0.58 0.60
WDs (↑) 0.59 0.54 0.59 0.61
WDa (↑) 0.65 0.63 0.66 0.68
Success (↑) 0.40 0.40 0.42 0.66
Relocate DTWs (↑) 0.09 0.20 0.52 0.52
DTWa (↑) 0.47 0.51 0.82 0.84
WDs (↑) 0.27 0.36 0.47 0.46
WDa (↑) 0.45 0.50 0.65 0.65
Success (↑) 0.01 0.00 0.20 0.07
Average DTWs (↑) 0.16 0.21 0.63 0.66
DTWa (↑) 0.49 0.52 0.82 0.84
WDs (↑) 0.32 0.38 0.64 0.64
WDa (↑) 0.45 0.48 0.74 0.76
Success (↑) 0.11 0.14 0.39 0.64
Table 1. Improvements with the IQL baseline on the Adroit benchmark. Bold denotes the best result.
 
Results with SAC Baseline
HiMAQ+SAC improves average DTWs from 0.56 to 0.63 and average success rate from 0.28 to 0.29, with notable trajectory similarity gains across all tasks.
Task Metric BC SAC MAQ+SAC HiMAQ+SAC
Door DTWs (↑) 0.18 -0.39 0.80 0.83
DTWa (↑) 0.42 -0.25 0.91 0.92
WDs (↑) 0.32 -0.28 0.71 0.75
WDa (↑) 0.41 -0.15 0.77 0.80
Success (↑) 0.02 0.43 0.56 0.50
Hammer DTWs (↑) -0.16 -1.10 0.61 0.64
DTWa (↑) 0.47 -0.33 0.91 0.92
WDs (↑) 0.11 -0.44 0.64 0.67
WDa (↑) 0.30 -0.19 0.78 0.82
Success (↑) 0.00 0.01 0.00 0.00
Pen DTWs (↑) 0.53 0.06 0.58 0.58
DTWa (↑) 0.58 -0.34 0.58 0.58
WDs (↑) 0.59 0.29 0.61 0.61
WDa (↑) 0.65 0.22 0.67 0.67
Success (↑) 0.40 0.32 0.41 0.51
Relocate DTWs (↑) 0.09 -0.55 0.25 0.46
DTWa (↑) 0.47 -0.10 0.66 0.80
WDs (↑) 0.27 -0.22 0.40 0.47
WDa (↑) 0.45 -0.05 0.61 0.66
Success (↑) 0.01 0.00 0.14 0.16
Average DTWs (↑) 0.16 -0.49 0.56 0.63
DTWa (↑) 0.49 -0.26 0.76 0.81
WDs (↑) 0.32 -0.17 0.59 0.63
WDa (↑) 0.45 -0.04 0.71 0.74
Success (↑) 0.11 0.19 0.28 0.29
Table 2.  Improvements with the SAC baseline on the Adroit benchmark. Bold denotes the best result.
 
Results with RLPD Baseline
HiMAQ+RLPD improves average DTWs from 0.56 to 0.62 and success rate from 0.52 to 0.56, consistently outperforming MAQ+RLPD on trajectory similarity while maintaining competitive task success.

Task Metric BC RLPD MAQ+RLPD HiMAQ+RLPD
Door DTWs (↑) 0.18 -0.06 0.76 0.83
DTWa (↑) 0.42 0.28 0.91 0.93
WDs (↑) 0.32 -0.14 0.71 0.76
WDa (↑) 0.41 0.10 0.76 0.79
Success (↑) 0.02 0.96 0.93 0.93
Hammer DTWs (↑) -0.16 -0.03 0.68 0.67
DTWa (↑) 0.47 0.37 0.94 0.94
WDs (↑) 0.11 -0.03 0.76 0.79
WDa (↑) 0.30 0.20 0.85 0.88
Success (↑) 0.00 1.00 0.56 0.58
Pen DTWs (↑) 0.53 0.48 0.54 0.61
DTWa (↑) 0.58 0.40 0.59 0.64
WDs (↑) 0.59 0.49 0.59 0.62
WDa (↑) 0.65 0.44 0.66 0.69
Success (↑) 0.40 0.62 0.42 0.61
Relocate DTWs (↑) 0.09 0.03 0.27 0.37
DTWa (↑) 0.47 0.32 0.69 0.74
WDs (↑) 0.27 0.02 0.38 0.42
WDa (↑) 0.45 0.20 0.55 0.60
Success (↑) 0.01 0.14 0.17 0.13
Average DTWs (↑) 0.16 0.10 0.56 0.62
DTWa (↑) 0.49 0.34 0.78 0.81
WDs (↑) 0.32 0.08 0.61 0.65
WDa (↑) 0.45 0.24 0.70 0.74
Success (↑) 0.11 0.68 0.52 0.56
Table 3.  Improvements with the RLPD baseline on the Adroit benchmark. Bold denotes the best result.
 
Impact of Macro Action Length
We study how macro action length H affects HiMAQ+RLPD, averaged over all four Adroit tasks. H = 9 achieves the best overall balance, with the highest success rate (0.56) and strong similarity scores (DTWs = 0.62, DTWa = 0.81, WDs = 0.65, WDa = 0.74). This optimal value matches MAQ+RLPD, confirming that the hierarchical extension does not alter the optimal macro action length. Performance improves from H = 1 to 9, then plateaus or declines.
method
Impact of different macro action lengths H in HiMAQ+RLPD.
Human Evaluations
To verify whether trajectory-level improvements are perceptually recognized as more human-like, we conducted human evaluations with 22 evaluators across two complementary tests.

Turing Test. Each evaluator saw 24 trials, each presenting two side-by-side video clips (one from a human demonstrator, one from an agent) and was asked to identify which was human. A higher win rate (fraction of trials where evaluators were fooled) indicates more human-like behavior. HiMAQ variants top the ranking:

HiMAQ+RLPD (43%) > HiMAQ+IQL (38%) > MAQ+RLPD (37%) > HiMAQ+SAC (34%) > MAQ+SAC (33%) > MAQ+IQL (32%) > RLPD (25%) > BC (24%) > SAC (18%) > IQL (13%).
method
Turing test win rates (fraction of trials where evaluators mistook an agent for a human).

Ranking Test. Each evaluator saw 21 pairs of agent videos and chose which appeared more human-like, producing pairwise win rates. The ranking by average win rate is:

Human (71%) > HiMAQ+RLPD (54%) > HiMAQ+SAC (52%) > HiMAQ+IQL (51%) > MAQ+RLPD (48%) > MAQ+SAC (38%) > MAQ+IQL (36%).

All three HiMAQ variants consistently rank above their MAQ counterparts, and HiMAQ+RLPD is the closest to the human baseline at 54% vs. 71%.


method
Human-likeness ranking test pairwise win rates. Each cell gives the fraction of trials in which Agent A (row) was judged more human-like than Agent B (column). The Avg column summarizes each agent’s overall score.
 

Conclusion

We present HiMAQ, a human-like RL framework that produces action trajectories consistent with human behaviors while optimizing rewards. HiMAQ hierarchically discretizes action sequences using a two-stage vector quantization hierarchy: the first stage discovers low-level subactions, while the second stage composes them into higher-level actions. Experiments on D4RL Adroit benchmarks demonstrate clear improvements over the non-hierarchical MAQ baseline, yielding more human-like behavior without sacrificing task performance. These benefits remain consistent across IQL, SAC, and RLPD. Human evaluations with 22 participants confirm that the quantitative trajectory improvements translate to perceptually recognizable human-likeness, with HiMAQ+RLPD achieving a Turing test win rate of 43% and ranking closest to real human demonstrations in the ranking test.
 

Citation


@article{nizamani2026enhancing,
  title={Enhancing Human-Likeness in Reinforcement Learning Agents via Hierarchical Macro Action Quantization},
  author={Nizamani, Usman and Luqman, M Shaheer and Fateh, Fawad Javed and Ali, Ali Shah and Popattia, Murad and Zia, M Zeeshan and Tran, Quoc-Huy},
  journal={arXiv preprint arXiv:2605.30928},
  year={2026}
}