Skin tissue is a complex heterogeneous material abundant with fibers. Various models capturing its anisotropy, nonlinearity, and viscoelasticity have been developed. However, the existence of multiple fiber families and their differences have been largely ignored. Furthermore, inhomogeneous deformation over the thickness is observed in the skin under shear deformation, which the traditional skin models do not predict. In this paper, we propose that two fiber families with distinct mechanical and structural properties exist in the skin within the framework of a general structure tensor-based constitutive strain energy model. Our constitutive model considers fiber families’ distinct properties and the consequent inhomogeneous deformation in the skin, showing good agreement with in vivo measurements of human face skin.