Octopi: Object Property Reasoning with Large Tactile-Language Models

Octopi: Object Property Reasoning with Large Tactile-Language Models, Samson Yu★, Kelvin Lin★, Anxing Xiao, Jiafei Duan, and Harold Soh★, Robotics: Science and Systems 2024 (R:SS 2024)
Links: Paper | Github | Project page

Check out our project page for more information!

Abstract: Physical intelligence is important for effective robot,manipulation. Recent work has investigated both vision and language modalities for physical reasoning; vision can reveal information about objects in the environment and language serves as an abstraction and communication medium for additional context. Although these works have demonstrated success on a variety of physical reasoning tasks, they are limited to physical properties that can be inferred from visual or language inputs. In this work, we investigate combining tactile perception with language, which enables embodied systems to obtain physical properties through interaction and apply common-sense reasoning.

We contribute a new dataset PhysiCLeAR, which comprises both physical/property reasoning tasks and annotated tactile videos obtained using a GelSight tactile sensor. We then introduce Octopi, a system that leverages both tactile representation learning and large vision-language models to predict and reason about tactile inputs with minimal language fine-tuning. Our evaluations on PhysiCLeAR show that Octopi is able to effectively use intermediate physical property predictions to improve physical reasoning in both trained tasks and for zero-shot reasoning.

Resources

You can find our paper here and check out our github!

Citation

Please consider citing our paper if you build upon our results and ideas.

Samson Yu★, Kelvin Lin★, Anxing Xiao, Jiafei Duan, and Harold Soh★, “Octopi: Object Property Reasoning with Large Tactile-Language Models”, Robotics: Science and Systems 2024 (R:SS 2024)

@article{yu2024octopi, title={Octopi: Object Property Reasoning with Large Tactile-Language Models}, author={Yu, Samson and Lin, Kelvin and Xiao, Anxing and Duan, Jiafei and Soh, Harold}, journal={arXiv preprint arXiv:2405.02794}, year={2024} }