Well, perhaps he just made a relevant example here? I mean, scientists are doing experiments on mammals (rats, rabbits, monkeys, dogs etc), testing medicine and physical reaction because most of these results can be extrapolated to humans. Observations on most gelded pets did no indicate any siginificant decline in physical endurance and strenght; gelded bulls are being succesfully employed on the field with optimal performance levels. That is relevant, in my opinion - after all, Premarin (Pregnant Mares Urine) is nr. 1 estrogen HRT substance - extracted from pregnant mares. Basically, it demonstrates how horse hormones are close to that of humans. And - some eunuch harem guards were known as fit, musculous guys;
http://www.rotten.com/library/sex/castration/eunuch/ (Warning, not very friendly site!)
And while a lessening testosterone (through loss of testicles) did not significantly lessen the eunuch's courage or his strength, it did however tend to make him calmer and more even-tempered. (Again, like the ox.)
testosterone in humans either is not neccesary to become an athlete or a body builder:
http://www.limbermen.com/Sandrak/index_menu.html#16 (8yr boy with normal voice pitch)
And Marsha Mar is right, pronounced fatigue in some eunuchs following castration is considered a temporary side-effect, very much like the hot flushes. It's also noted on the GNRH-antagonist (wich I am familiar with) manuals and usually vanishes after a few months. That is, malignant fatigue wanes off. Apparently you won't have that male surplus of extra strenght you could before castration. Just that you'll remain as fit or more than a sporty woman, with correct approach.
Although castration result in a lower metabolic rate, it may either cause individual weight loss or gain due to changes in insulin, IGF-1 and glucose metabolism, affecting your appetite (mine was reduced). You'll need to observe the changes and adjust your meals accordingly.