In the United States, about one out of eight women has been diagnosed with breast cancer. This year, about 200,000 new cases of invasive breast cancer are expected to be diagnosed in the U.S. Radiation therapy is a standard procedure to treat breast cancer. During the radiation therapy, the high energy penetration radiation is used to destroy the cancer cells. However, radiation damages normal cells as well, and any movement of the patient’s body during the treatment will expose more normal cells to the radiation. In order to reduce the unnecessary radiation to normal cells, a real-time infrared (IR) thermal image-based tumor tracking system for breast cancer radiotherapy is developed in this research. In the system, the real-time feedback of breast tumor locations consists of the images taken by an IR thermal camera system. A serial six-degree-of-freedom manipulator is used to realize the real-time position and orientation adjustment of the X-ray radiation head (i.e., a linear accelerator). There are several advantages of this system: (i) IR radiation is not harmful to human body. (ii) Quick image acquisition can be achieved. Faster control speed can lead to less radiation on normal cells through the quick response of the system if the radiation head is away from the target by some reason. All these will increase the radiation effectiveness. With the reduced radiation to the patients, the therapy can be safer and more protective than the traditional methods.