Now scientists believe they have uncovered the reason why - and it lies in the way our brains develop as we age.
They say it gets harder to focus without being distracted past the age of 55, particularly when under stress, because of the way our brains change over time.
A study has found stress has much less of an effect on young people, who can focus on the task at hand and block out unnecessary distractions. By contrast, scans showed older individuals lose these skills.
A study, led by the University of South Carolina, put participants aged 55 to 75 in a stressful situation then asked them to pick the clearest of two black and white photographs.
When the results were compared to those of a group aged 18 to 34, the older group was worse at focusing, taking longer to find the answer. MRI scans found older adults under stress showed less activity in that part of the brain that enables us to pay attention and ignore competing thoughts and distractions. Younger adults, by contrast, saw no difference.
Professor Mara Mather, a co-author of the study, said: "Trying hard to complete a task increases emotional arousal, so when younger adults try hard, this should increase their ability to ignore distracting information.
"But for older adults, trying hard may make both what they are trying to focus on and other information stand out more."
The experiment enlisted 24 participants in the older group and 28 in the younger group. Their task, repeated 160 times, was to identify the clearest image out of two pictures of a building and an object. The correct answer flashed up for just a tenth of a second. To place extra pressure on them, participants were threatened with electric shocks. The results, published in Nature Human Behaviour, found younger people answered an average of 143 milliseconds faster. Authors think this is because part of the brain which controls the ability to focus under pressure appears to weaken with age.
The pathways between this area and the parts of the brain involved in looking at images of places and controlling what we pay attention to and what we ignore showed less activity in older adults.
These areas are also linked to Alzheimer’s disease and other forms of dementia.
Mather said: "Deciphering exactly how these changes in the brain occur as we age could one day help us uncover how to protect the brain from cognitive decline."Daily Mail