Empathy can help designers discover and fulfill users’ latent needs in design, which can lead to better design solutions. To achieve deep user understanding, designers utilize their ability to empathize with users. However, empathy is always incomplete and inconstant in user understanding. This study applies a quantitative behavioral empathy measure to a case of designers attempting to understand the needs of users of a future semi-autonomous vehicle technology. In total, 16 designers and 48 users are involved. We study how interpersonal differences between designers and users and different users’ mental content impact the level of empathic understanding. Specifically, these factors are national cultural differences, visiting experience, binary gender differences, the relationship between users and designers, and user mental content types. The results show that cultural differences and visiting experiences significantly affect designers’ empathic accuracy and emotional tone accuracy in both physical and online interviews. These influences vary depending on contextual factors that are related to the participant's own experience and the experiment itself.