Smart Gripper with Triboelectric Sensors and Data-driven Machine Learning for Object Classification

Abstract

Reliable material recognition during robotic grasping remains challenging when visual perception is degraded by poor illumination, occlusion, or similarity in object appearance, such as in the case of glass and plastic. To address this limitation, this thesis presents a visuo–tactile robotic perception framework that combines self-powered triboelectric tactile sensing with RGB vision for material classification at contact. A compact two-finger robotic gripper with embedded single-electrode triboelectric nanogenerator (TENG) sensing elements was designed, fabricated, and integrated with a UR10 robotic manipulator. The finger sensors were made from soft silicon-based flexible materials containing carbon-fiber cloth electrodes which can capture the contact generated tribo-electric signals directly while the robot is grasping an object. Preprocessing of the data captured by the tactile sensor was done and presented in two types (one-dimensional or linear; and two-dimensional or spatial) formats so they could be classified based on learning algorithms. Simultaneously, the visual component was constructed by applying a faster R-CNN model with resnet-50 FPN base to recognize objects visually, and to determine the multi-modal output laterally. An evaluation of this framework was conducted on six different classes of materials - metal, glass, plastic, paper, ceramic and organic, using a rigidly controlled robotic grasping protocol over two light intensities: regular light and reduced light. Results indicated that the tactile branch was relatively stable and consistent when visual performance decreased due to lower illumination. From a model perspective, the best unimodal classification performance was achieved by the two-dimensional representation of tactile data demonstrating that designing representations of tactile data had significant impacts on accuracy of recognition. From a systems perspective, the multimodal configuration produced the highest cumulative prediction success rates compared to either a tactile-only or vision-only configuration. Therefore, these results demonstrated that the tactile sensing via triboelectrics provides relevant contact-related information that is complementary to RGB vision and therefore enhances the ability of robots to perform material identification in manipulation tasks. This work provided a practical and cost-effective way for developing visuo-tactile perception frameworks and will support further development of self-sustaining tactile sensors for material aware robotic functions.