Phone usage while driving is unanimously considered a really dangerous habit due to a strong correlation with road accidents. This paper proposes a phone-use monitoring system that detects the driver’s handheld phone use and eliminates the distraction at once. Specifically, the proposed system emits periodic ultrasonic pulses to sense if the phone is being held in hand or placed on support surfaces (e.g., seat and cup holder) by capturing the unique signal interference resulted from the contact object’s damping, reflection and refraction. We derive the short-time Fourier transform from the microphone data to describe such impacts and develop a CNN-based binary classifier to discriminate the phone use between the handheld and the handsfree status. Additionally, we design a classification error correction filter to correct the classification errors during the monitoring. The experiments with six people, one phone and one car model show that our system achieves 99% accuracy in recognizing handheld phone-use activities.