FasTac: A Curved Multispectral Vision-Based Tactile Sensor for High-Speed High-Precision 3D Shape and Force Perception
AuthorsXiaofan Lu, Kaiji Huang, Jiahui Chen, Yuankai Lin, Hua Yang, Zhouping Yin
Resources
FasTac is a fast curved tactile fingertip that uses multispectral vision and FPGA processing to sense 3D contact shape and forces during dexterous robot manipulation.
Key results
RGB-NIR reconstruction with the boundary prior achieves 0.0415 mm mean absolute error.
HyperForce normal-force estimation error.
HyperForce shear-force estimation error.
Image-to-normal-force latency in milliseconds on the Xilinx Zynq UltraScale+ FPGA.
Energy consumed per frame by the complete FPGA pipeline in mJ/frame.
Excitation frequency recovered by the FPGA during dynamic tactile sensing, in Hz.
What the paper found
Researchers Xiaofan Lu, Kaiji Huang, Jiahui Chen, Yuankai Lin, Hua Yang, and Zhouping Yin at Huazhong University of Science and Technology present FasTac, a compact curved tactile fingertip for simultaneous 3D shape, three-axis force, and dynamic contact perception. Its key innovation is a single OmniVision OV2736 RGB-IR CMOS sensor that captures spatially aligned RGB and near-infrared images in one exposure. The additional NIR channel makes photometric stereo over curved elastomers more stable, while a CAD-derived boundary prior enables fast Poisson depth reconstruction without curvature-induced drift. For force sensing, HyperForce combines reconstructed normal displacement with marker-derived tangential displacement and uses position-aware dynamic convolution to approximate the curved elastomer’s spatially varying stiffness. The resulting depth mean absolute error is 0.0415 mm, and force normalized mean absolute error reaches 2.74% for normal force and 2.39% for shear force. The complete image-to-normal-force pipeline runs on a Xilinx Zynq UltraScale+ FPGA in 1.09 ms, consuming 8.41 mJ/frame, compared with 3.26 ms on an NVIDIA RTX 4060 GPU. Experiments include strawberry, fingerprint, and LED-array reconstruction, friction-aware bottle grasping, and vibration tracking; the FPGA correctly preserves a 100.04 Hz excitation that CPU and GPU pipelines fail to reproduce reliably. The work demonstrates an edge-deployed tactile sensor designed for high-speed, high-precision dexterous manipulation.
Original abstract
Curved tactile fingertips for dexterous manipulation must resolve fine contact geometry, distinguish normal and tangential loads, and capture transient signals. Existing curved vision-based tactile sensors struggle to combine accurate 3D reconstruction, three-axis force estimation, and high-speed processing in a compact form. This article presents FasTac, a curved vision-based tactile sensor integrating multispectral photometric stereo, dynamic-convolution force estimation, and hardware acceleration on a field-programmable gate array (FPGA). Single-image-sensor simultaneous multispectral imaging provides spatially aligned observations for robust surface normal estimation, followed by boundary-prior fast Poisson depth reconstruction. HyperForce uses position-aware dynamic convolution to model the spatially nonuniform mechanical response of curved elastomers and estimate three-axis forces. The complete image-to-normal-force pipeline is deployed on an FPGA. Experiments show that near-infrared (NIR) illumination and the boundary prior decrease depth mean absolute error (MAE) from 0.2730 mm to 0.0415 mm; HyperForce achieves normalized mean absolute error (NMAE) values of 2.74% and 2.39% for normal and shear forces, respectively; and FPGA deployment shortens processing latency from 3.26 ms on the GPU to 1.09 ms. Multi-object reconstruction, feedback grasping, and vibration measurement validate fine geometric perception, stable force feedback, and dynamic contact sensing.
Read the original paperMore in Robotics
Browse all 50 papers →JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation
Chuyang Xiao, Peilin Meng, David Held
JAMB helps two robot arms coordinate by jointly imagining their future movements and the changing 3D scene before acting.
Rolling-WAM: World Action Models with Rolling Imagination
Yinghua Zhou, Junjie Ye, Yiqi Zhao, Hao Dong, Celina Shiyu Wang, Ruohai Ge, Tingyi Yang, Basile Van Hoorick, Gaurav Sukhatme, Vitor Guizilini, Yue Wang
Rolling-WAM keeps future robot actions partially imagined and refined over time, making world-model-based manipulation replan 4.5 times faster.
Training-free Behavior Cloning
Maximilian Adang, Timothy Chen, Lars Osterberg, Aiden Swann, Mac Schwager
A fast, training-free robot controller reuses and corrects demonstration trajectories to deliver traceable behavior at real-time speeds.