Standard disclaimer: This really is a comment, but it's too long, so...
My gut feeling is that the efficiency of catching duplicates has been going down. When I started here, the site was about a year old, and the old-timers could largely rely on memory in spotting duplicates (exact or abstract), and move swiftly. I managed to build a similar personal smallish database, and join in the battle. I would often remember who had given a good answer to an essentially duplicate question, search his/her history of answers, and act.
Since then the number of questions has exploded, some of the prominent old-timers have disappeared. Recently a top user described his approach as: "This question is probably a duplicate, but retyping an answer is faster than locating the original, so..."
Such a comment could be taken as a sign of lazy attitude, but I want to make it clear that I'm not pointing fingers. As the size of the site grows, a point will be reached, where no one can remember for sure. In other words that comment gains more validity each day. Each and every one of us will eventually reach a point, where personal recollection will no longer help. The break even -point varies from person to person, but we all have one.
Younger members may have grown up with search engines, and know how to use them to optimal effect. Consequently their break even point may be way higher. If somebody wants to share effective search tips and pointers, I'm all ears.
BECAUSE I think that we should and could do a better job in spotting duplicates than we do currently.
TL; DR; The problem is somewhat caused by the deadly sin of sloth. On the part of the askers, also on the part of the FGITW answerers,... But also by understandable human limitations.
CAVEAT: I largely ignore calculus and algebra-precalculus, so I may not have seen the worst of it. The ultra-high volume tags are likely more difficult to manage in terms of catching (abstract) duplicates than my favorite ones.
Addition on July 27, 2016 (the only part of this post written after I was elected)
There's no end to the sloth. I don't see any alternatives to downvoting answers by veteran users to questions that they surely know have been handled on the site many times. If not in exact same form then essentially so. I am more forgiving to relative newcomers, but the ultra high rep users should IMHO be role models here.
I am not talking about the "accidental" duplicates, some question that may have been asked twice or thrice. This is about the oft recurring questions, all parts of which have reappeared on the site several times. Questions that a veteran cannot fail to think is a dup.