Experimental Results
We evaluate HiMAQ on four dexterous manipulation tasks from D4RL’s Adroit benchmark — Door, Hammer, Pen, and Relocate — each controlled by a 24-DOF simulated hand with 25 human demonstration trajectories. We compare against BC, vanilla RL baselines (IQL, SAC, RLPD), and their MAQ-augmented counterparts. Human-likeness is measured by Dynamic Time Warping (DTW) and Wasserstein Distance (WD) on both states and actions; higher normalized scores indicate more human-like behavior.
Results with IQL Baseline
HiMAQ+IQL improves average DTWs from 0.63 to 0.66 and success rate from 0.39 to 0.64, with a particularly large gain on the Hammer task (success rate from 0.00 to 0.87).
| Task |
Metric |
BC |
IQL |
MAQ+IQL |
HiMAQ+IQL |
| Door |
DTWs (↑) |
0.18 |
0.43 |
0.84 |
0.87 |
| DTWa (↑) |
0.42 |
0.61 |
0.95 |
0.96 |
| WDs (↑) |
0.32 |
0.48 |
0.75 |
0.77 |
| WDa (↑) |
0.41 |
0.50 |
0.81 |
0.82 |
| Success (↑) |
0.02 |
0.16 |
0.93 |
0.95 |
| Hammer |
DTWs (↑) |
-0.16 |
-0.14 |
0.64 |
0.68 |
| DTWa (↑) |
0.47 |
0.45 |
0.92 |
0.94 |
| WDs (↑) |
0.11 |
0.12 |
0.75 |
0.73 |
| WDa (↑) |
0.30 |
0.30 |
0.84 |
0.87 |
| Success (↑) |
0.00 |
0.01 |
0.00 |
0.87 |
| Pen |
DTWs (↑) |
0.53 |
0.34 |
0.55 |
0.57 |
| DTWa (↑) |
0.58 |
0.51 |
0.58 |
0.60 |
| WDs (↑) |
0.59 |
0.54 |
0.59 |
0.61 |
| WDa (↑) |
0.65 |
0.63 |
0.66 |
0.68 |
| Success (↑) |
0.40 |
0.40 |
0.42 |
0.66 |
| Relocate |
DTWs (↑) |
0.09 |
0.20 |
0.52 |
0.52 |
| DTWa (↑) |
0.47 |
0.51 |
0.82 |
0.84 |
| WDs (↑) |
0.27 |
0.36 |
0.47 |
0.46 |
| WDa (↑) |
0.45 |
0.50 |
0.65 |
0.65 |
| Success (↑) |
0.01 |
0.00 |
0.20 |
0.07 |
| Average |
DTWs (↑) |
0.16 |
0.21 |
0.63 |
0.66 |
| DTWa (↑) |
0.49 |
0.52 |
0.82 |
0.84 |
| WDs (↑) |
0.32 |
0.38 |
0.64 |
0.64 |
| WDa (↑) |
0.45 |
0.48 |
0.74 |
0.76 |
| Success (↑) |
0.11 |
0.14 |
0.39 |
0.64 |
Table 1. Improvements with the IQL baseline on the Adroit benchmark. Bold denotes the best result.
Results with SAC Baseline
HiMAQ+SAC improves average DTWs from 0.56 to 0.63 and average success rate from 0.28 to 0.29, with notable trajectory similarity gains across all tasks.
| Task |
Metric |
BC |
SAC |
MAQ+SAC |
HiMAQ+SAC |
| Door |
DTWs (↑) |
0.18 |
-0.39 |
0.80 |
0.83 |
| DTWa (↑) |
0.42 |
-0.25 |
0.91 |
0.92 |
| WDs (↑) |
0.32 |
-0.28 |
0.71 |
0.75 |
| WDa (↑) |
0.41 |
-0.15 |
0.77 |
0.80 |
| Success (↑) |
0.02 |
0.43 |
0.56 |
0.50 |
| Hammer |
DTWs (↑) |
-0.16 |
-1.10 |
0.61 |
0.64 |
| DTWa (↑) |
0.47 |
-0.33 |
0.91 |
0.92 |
| WDs (↑) |
0.11 |
-0.44 |
0.64 |
0.67 |
| WDa (↑) |
0.30 |
-0.19 |
0.78 |
0.82 |
| Success (↑) |
0.00 |
0.01 |
0.00 |
0.00 |
| Pen |
DTWs (↑) |
0.53 |
0.06 |
0.58 |
0.58 |
| DTWa (↑) |
0.58 |
-0.34 |
0.58 |
0.58 |
| WDs (↑) |
0.59 |
0.29 |
0.61 |
0.61 |
| WDa (↑) |
0.65 |
0.22 |
0.67 |
0.67 |
| Success (↑) |
0.40 |
0.32 |
0.41 |
0.51 |
| Relocate |
DTWs (↑) |
0.09 |
-0.55 |
0.25 |
0.46 |
| DTWa (↑) |
0.47 |
-0.10 |
0.66 |
0.80 |
| WDs (↑) |
0.27 |
-0.22 |
0.40 |
0.47 |
| WDa (↑) |
0.45 |
-0.05 |
0.61 |
0.66 |
| Success (↑) |
0.01 |
0.00 |
0.14 |
0.16 |
| Average |
DTWs (↑) |
0.16 |
-0.49 |
0.56 |
0.63 |
| DTWa (↑) |
0.49 |
-0.26 |
0.76 |
0.81 |
| WDs (↑) |
0.32 |
-0.17 |
0.59 |
0.63 |
| WDa (↑) |
0.45 |
-0.04 |
0.71 |
0.74 |
| Success (↑) |
0.11 |
0.19 |
0.28 |
0.29 |
Table 2. Improvements with the SAC baseline on the Adroit benchmark. Bold denotes the best result.
Results with RLPD Baseline
HiMAQ+RLPD improves average DTWs from 0.56 to 0.62 and success rate from 0.52 to 0.56, consistently outperforming MAQ+RLPD on trajectory similarity while maintaining competitive task success.
| Task |
Metric |
BC |
RLPD |
MAQ+RLPD |
HiMAQ+RLPD |
| Door |
DTWs (↑) |
0.18 |
-0.06 |
0.76 |
0.83 |
| DTWa (↑) |
0.42 |
0.28 |
0.91 |
0.93 |
| WDs (↑) |
0.32 |
-0.14 |
0.71 |
0.76 |
| WDa (↑) |
0.41 |
0.10 |
0.76 |
0.79 |
| Success (↑) |
0.02 |
0.96 |
0.93 |
0.93 |
| Hammer |
DTWs (↑) |
-0.16 |
-0.03 |
0.68 |
0.67 |
| DTWa (↑) |
0.47 |
0.37 |
0.94 |
0.94 |
| WDs (↑) |
0.11 |
-0.03 |
0.76 |
0.79 |
| WDa (↑) |
0.30 |
0.20 |
0.85 |
0.88 |
| Success (↑) |
0.00 |
1.00 |
0.56 |
0.58 |
| Pen |
DTWs (↑) |
0.53 |
0.48 |
0.54 |
0.61 |
| DTWa (↑) |
0.58 |
0.40 |
0.59 |
0.64 |
| WDs (↑) |
0.59 |
0.49 |
0.59 |
0.62 |
| WDa (↑) |
0.65 |
0.44 |
0.66 |
0.69 |
| Success (↑) |
0.40 |
0.62 |
0.42 |
0.61 |
| Relocate |
DTWs (↑) |
0.09 |
0.03 |
0.27 |
0.37 |
| DTWa (↑) |
0.47 |
0.32 |
0.69 |
0.74 |
| WDs (↑) |
0.27 |
0.02 |
0.38 |
0.42 |
| WDa (↑) |
0.45 |
0.20 |
0.55 |
0.60 |
| Success (↑) |
0.01 |
0.14 |
0.17 |
0.13 |
| Average |
DTWs (↑) |
0.16 |
0.10 |
0.56 |
0.62 |
| DTWa (↑) |
0.49 |
0.34 |
0.78 |
0.81 |
| WDs (↑) |
0.32 |
0.08 |
0.61 |
0.65 |
| WDa (↑) |
0.45 |
0.24 |
0.70 |
0.74 |
| Success (↑) |
0.11 |
0.68 |
0.52 |
0.56 |
Table 3. Improvements with the RLPD baseline on the Adroit benchmark. Bold denotes the best result.
Impact of Macro Action Length
We study how macro action length H affects HiMAQ+RLPD, averaged over all four Adroit tasks. H = 9 achieves the best overall balance, with the highest success rate (0.56) and strong similarity scores (DTWs = 0.62, DTWa = 0.81, WDs = 0.65, WDa = 0.74). This optimal value matches MAQ+RLPD, confirming that the hierarchical extension does not alter the optimal macro action length. Performance improves from H = 1 to 9, then plateaus or declines.
Impact of different macro action lengths H in HiMAQ+RLPD.
Human Evaluations
To verify whether trajectory-level improvements are perceptually recognized as more human-like, we conducted human evaluations with 22 evaluators across two complementary tests.
Turing Test. Each evaluator saw 24 trials, each presenting two side-by-side video clips (one from a human demonstrator, one from an agent) and was asked to identify which was human. A higher win rate (fraction of trials where evaluators were fooled) indicates more human-like behavior. HiMAQ variants top the ranking:
HiMAQ+RLPD (43%) > HiMAQ+IQL (38%) > MAQ+RLPD (37%) > HiMAQ+SAC (34%) > MAQ+SAC (33%) > MAQ+IQL (32%) > RLPD (25%) > BC (24%) > SAC (18%) > IQL (13%).

Turing test win rates (fraction of trials where evaluators mistook an agent for a human).
Ranking Test. Each evaluator saw 21 pairs of agent videos and chose which appeared more human-like, producing pairwise win rates. The ranking by average win rate is:
Human (71%) > HiMAQ+RLPD (54%) > HiMAQ+SAC (52%) > HiMAQ+IQL (51%) > MAQ+RLPD (48%) > MAQ+SAC (38%) > MAQ+IQL (36%).
All three HiMAQ variants consistently rank above their MAQ counterparts, and HiMAQ+RLPD is the closest to the human baseline at 54% vs. 71%.
Human-likeness ranking test pairwise win rates. Each cell gives the fraction of trials in which Agent A (row) was judged more human-like than Agent B (column). The Avg column summarizes each agent’s overall score.
Conclusion
We present HiMAQ, a human-like RL framework that produces action trajectories consistent with human behaviors while optimizing rewards. HiMAQ hierarchically discretizes action sequences using a two-stage vector quantization hierarchy: the first stage discovers low-level subactions, while the second stage composes them into higher-level actions. Experiments on D4RL Adroit benchmarks demonstrate clear improvements over the non-hierarchical MAQ baseline, yielding more human-like behavior without sacrificing task performance. These benefits remain consistent across IQL, SAC, and RLPD. Human evaluations with 22 participants confirm that the quantitative trajectory improvements translate to perceptually recognizable human-likeness, with HiMAQ+RLPD achieving a Turing test win rate of 43% and ranking closest to real human demonstrations in the ranking test.