The heat transfer characteristics and kinetics of Camellia oleifera seeds under hot-air drying were investigated at different temperatures (40, 60, and 80 °C) and loading densities (0.92, 1.22, and 1.52 g/cm2) with a constant air velocity of 1 m/s. Twelve common drying kinetic models were selected to fit the experimental data. The most suitable model was chosen to describe the hot-air drying process of C. oleifera seeds and help in its optimization. The results showed that the drying temperature has a significant influence on the hot-air drying characteristics of C. oleifera seeds. As the drying air temperature increases, the drying time decreases. The effect of the loading density on the drying characteristics of C. oleifera seeds is much smaller than that of temperature. With the increase in the loading density, the drying time slightly increases. The hot-air drying curve of C. oleifera seeds consists of a very short acceleration rate period at the beginning and a long falling rate period, indicating that the drying of C. oleifera seeds is mainly controlled by the diffusion of moisture inside the material. An effective moisture diffusion coefficient of C. oleifera seeds was estimated to range from 0.81256 × 10−9 to 3.28496 × 10−9 m2/s within the temperature range studied. The average activation energy was 28.27979 kJ/mol. The logarithmic model was found to be the best model to describe the kinetics of hot-air drying of C. oleifera seeds.