Imitation Learning with Limited Actions via Diffusion Planners and Deep Koopman Controllers, Jianxin Bi★, Kelvin Lin★, Kaiqi Chen★, Yifei Huang★ and Harold Soh★, IEEE International Conference on Robotics and Automation (ICRA)
Links:
Abstract: Recent advances in diffusion-based robot policies have demonstrated significant potential in imitating multi-modal behaviors. However, these approaches typically require large quantities of demonstration data paired with corresponding robot action labels, creating a substantial data collection burden. In this work, we propose a plan-then-control framework aimed at improving the action-data efficiency of inverse dynamics controllers by leveraging observational demonstration data. Specifically, we adopt a Deep Koopman Operator framework to model the dynamical system and utilize observation-only trajectories to learn a latent action representation. This latent representation can then be effectively mapped to real high-dimensional continuous actions using a linear action decoder, requiring minimal action-labeled data. Through experiments on simulated robot manipulation tasks and a real robot experiment with multi-modal expert demonstrations, we demonstrate that our approach significantly enhances action-data efficiency and achieves high task success rates with limited action data.
Resources
You can find our paper here and check out our github!
Citation
Please consider citing our paper if you build upon our results and ideas.
Jianxin Bi★, Kelvin Lin★, Kaiqi Chen★, Yifei Huang★ and Harold Soh★, “Imitation Learning with Limited Actions via Diffusion Planners and Deep Koopman Controllers”, IEEE International Conference on Robotics and Automation (ICRA)
@inproceedings{bi2025imitationlearninglimitedactions, title = {Imitation Learning with Limited Actions via Diffusion Planners and Deep Koopman Controllers}, author = {Bi, Jianxin and Lim, Kelvin and Chen, Kaiqi and Huang, Yifei and Soh, Harold}, booktitle = {2025 IEEE International Conference on Robotics and Automation (ICRA)}, year = {2024}, eprint = {2410.07584}, archiveprefix = {arXiv}, primaryclass = {cs.RO}, url = {https://arxiv.org/abs/2410.07584}, }
Contact
If you have questions or comments, please contact Jianxin or Harold.
