Australian psychiatrist John J. McGrath and colleagues found that children of older fathers were likely to have lower IQs than children of younger fathers. Their kids were also more likely to be schizophrenic or autistic.
The children of 50 year old fathers had average IQs 6 points lower than the children of 20 year old fathers. That is half a standard deviation - a pretty substantial difference.
A woman has all her eggs at birth. They run the risk of wearing out and breaking down, which is one of the reasons that children of older mothers are more likely to have some birth defects. However, the children of older mothers did not score any lower on the intelligence measures in this study.
Sperm, on the other hand, and made continuously. It was thought that this prevented the kind of breakdown with age that eggs show. McGrath and colleagues, though, point out that there are many more steps in the copying process to produce sperm in older fathers. The sperm of 20-year-old men are the result of about 150 cell divisions; those of 50-year-old men, 840. Each division provides a new opportunity for mutation and error.