Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization

Umer Ahmed^*, Syed Ahmed Mahmood^*, Fawad Javed Fateh, M. Shaheer Luqman, M. Zeeshan Zia, Quoc-Huy Tran

^* indicates joint first author.

Abstract

We propose a novel hierarchical spatiotemporal vector quantization framework for unsupervised skeleton-based temporal action segmentation. We first introduce a hierarchical approach, which includes two consecutive levels of vector quantization. Specifically, the lower level associates skeletons with fine-grained subactions, while the higher level further aggregates subactions into action-level representations. Our hierarchical approach outperforms the non-hierarchical baseline, while primarily exploiting spatial cues by reconstructing input skeletons. Next, we extend our approach by leveraging both spatial and temporal information, yielding a hierarchical spatiotemporal vector quantization scheme. In particular, our hierarchical spatiotemporal approach performs multi-level clustering, while simultaneously recovering input skeletons and their corresponding timestamps. Lastly, extensive experiments on multiple benchmarks, including HuGaDB, LARa, and BABEL, demonstrate that our approach establishes a new state-of-the-art performance and reduces segment length bias in unsupervised skeleton-based temporal action segmentation.

Motivation

Unsupervised skeleton-based temporal action segmentation remains a challenging problem due to the complex spatiotemporal dynamics of human motion and the absence of reliable annotations. While skeleton data provides a structured representation of human movement, existing approaches often struggle to capture both fine-grained sub-actions and higher-level action semantics simultaneously. Many prior methods rely on flat representations or single-level clustering, limiting their ability to model hierarchical action structures inherent in real-world activities. Additionally, methods that incorporate temporal modeling often require careful balancing of multiple objectives or fail to effectively disentangle spatial and temporal information. These limitations motivate the need for a unified framework that can hierarchically model actions while jointly leveraging spatial structure and temporal evolution.

Approach

We introduce HiST-VQ, a hierarchical spatiotemporal vector quantization approach for unsupervised skeleton-based temporal action segmentation. Instead of performing flat clustering over motion embeddings, HiST-VQ structures the clustering process across multiple levels, enabling it to discover short-term motion units that group together to form cohesive human actions. Furthermore, it jointly reconstructs the input skeletons together with timestamps, incorporating explicit temporal modeling. This allows the learned representations to capture both structural pose information and temporal progression within long sequences. Our model achieves state-of-the-art performance and reduces segment length bias (i.e., prior works tend to predict long segments and fail to capture short segments) than previous methods in unsupervised skeleton-based temporal action segmentation.

Overview. (a) Previous unsupervised skeleton-based temporal action segmentation methods, e.g., SMQ, rely on traditional vector quantization techniques, which perform flat clustering and mostly exploit spatial cues via reconstructing input skeletons. (b) We propose a hierarchical spatiotemporal vector quantization approach, which conducts multi-level clustering and exploits both spatial and temporal cues by jointly reconstructing input skeletons and their associated timestamps. Our approach obtains not only better accuracy but also less bias in predicted segment lengths. In the above figures, output skeletons with the same color belong to the same action.

Method. Given an input skeleton sequence S with associated timestamps T, we pass S to an encoder, which maps each joint sequence independently to the latent space. We concatenate the embedded joint sequences into the embedded skeleton sequence and divide it into patches along the temporal dimension, yielding X^P. Next, each patch p_k is first assigned to the nearest sub-action prototype z_j, which is then assigned to the closest action prototype a_i. Lastly, quantized action patches Q_A are passed through a spatial decoder to reconstruct the input skeleton sequence, while quantized subaction patches Q_Z are fed to a temporal decoder to recover the input timestamps. The encoder, decoders, and prototypes are learned jointly by optimizing a combination of spatial reconstruction loss, temporal reconstruction loss, and commitment losses.

Experimental Results

We evaluate our HiST-VQ model on 3 publicly available skeleton-based action segmentation datasets. Namely, HuGaDB, LARa and BABEL.

Comparisons on HuGaDB and LARa

We evaluate HiST-VQ on two standard skeleton-based benchmarks, HuGaDB and LARa, and compare it with existing unsupervised methods. Across both datasets, HiST-VQ consistently achieves superior performance, demonstrating the effectiveness of its hierarchical and spatiotemporal design. Notably, the model captures fine-grained motion patterns while maintaining coherent higher-level segmentation, leading to improvements in overall segmentation quality. Furthermore, the integration of temporal modeling enhances boundary detection and reduces fragmentation in predicted segments. These results highlight the advantage of jointly learning hierarchical representations over traditional flat approaches.

Comparisons on HuGaDB and LARa. Bold and underline denote the best and second best respectively.

Comparisons on BABEL Subsets

We further evaluate HiST-VQ on multiple subsets of the BABEL dataset, which introduces significant challenges due to diverse motion patterns and long temporal sequences. HiST-VQ consistently outperforms prior methods across all subsets, demonstrating strong generalization to complex and varied data distributions. The hierarchical structure enables effective modeling of compositional actions, while temporal modeling improves the consistency of segment transitions. As a result, HiST-VQ produces more accurate and stable segmentations compared to existing approaches. These findings confirm the robustness and scalability of the proposed framework.

Comparisons on BABEL subsets. Bold and underline denote the best and second best respectively.

Segment Length Bias Comparisons

We analyze the segment length bias of HiST-VQ in comparison to existing unsupervised segmentation methods. Prior approaches often produce segments with uniform or biased lengths, limiting their ability to reflect true action durations. In contrast, HiST-VQ reduces this bias by leveraging hierarchical representations and spatiotemporal modeling. The multi-level quantization allows the model to adapt to varying action durations, while temporal cues improve alignment with motion dynamics. Consequently, HiST-VQ generates more diverse and realistic segment lengths, leading to improved overall segmentation performance.

Segment Length Bias Comparisons. Bold and underline denote the best and second best respectively.

Histograms of segment lengths on BABEL Subset-3.

Qualitative Results

Qualitative results on HuGaDB, LARa, BABAL Subset-2, and BABEL Subset-3. Colored segments represent predicted actions with a particular color denoting the same action across all models.

Citation


@article{ahmed2026unsupervised,
  title={Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization},
  author={Ahmed, Umer and Mahmood, Syed Ahmed and Fateh, Fawad Javed and Luqman, M Shaheer and Zia, M Zeeshan and Tran, Quoc-Huy},
  journal={arXiv preprint arXiv:2604.15196},
  year={2026}
}