Smartphones continue to proliferate throughout our daily lives, not only in sheer quantity but also in their ever-growing list of uses. In addition to communication and entertainment, smartphones can also be used as a credit card to make a contactless payment on Kiosk Systems, such as ordering food, printing tickets and self-checkout. When a user holds the phone close to the Kiosk system to present payment credentials, we propose to also verify the user’s identity based on a photo of the back of their smartphone gripping hand, which provides a second security layer. Compared to the widely used facial recognition, the proposed approach addresses the recent struggles of identifying faces under masks and the public concerns of potential privacy erosion, racial bias and misuse. We find that the geometry of each individual’s hand, when it grips a phone, is identifiable and then design a vision-based approach to extract the gripping hand biometrics. In particular, we develop hand image processing schemes to detect and localize the gripping hand while denoising and normalizing the hand images (e.g., size and color). Furthermore, we develop a Convolutional Neural Network (CNN)-based algorithm to distinguish smartphone users’ gripping hand images for authentication. Experiments with 20 participants show that the system achieves 99.5% accuracy for user verification.