Objective: To test the hypothesis that white matter integrity, as measured by diffusion tensor and magnetization transfer MRI is significantly associated with cognitive ability measured in youth and old age. Methods: Forty, nondemented, surviving participants of the Scottish Mental Survey of 1932 underwent brain MRI and a battery of psychometric tests covering major cognitive domains and tests of information processing efficiency. IQ scores were available from age 11. Mean diffusivity, fractional anisotropy (FA), and magnetization transfer ratio (MTR) were measured in frontal and parieto-occipital white matter and centrum semiovale. Results: Centrum semiovale FA correlated (r = 0.36 to 0.56; p < 0.02) with contemporaneous (age 83) scores on psychometric tests of nonverbal reasoning, working memory, executive function, and information processing efficiency. Centrum semiovale FA also correlated with IQ at age 11 (r = 0.37; p = 0.02). Controlling for IQ at age 11 and information processing at age 83 attenuated the association between centrum semiovale FA and general cognitive ability by approximately 85%. MTR, largely, did not show significant correlations with cognitive test scores. Conclusions: These data support the information processing efficiency hypothesis of cognitive aging and suggest one foundation for individual differences in processing efficiency. They also suggest that studies of imaging and cognition in the elderly should take into account prior mental ability rather than assuming that any associations between imaging parameters and cognitive test scores are the result of age-related changes.