DH: You're right, I guess if you have one card that is dramatically older than the others, closing it could have a noticeable impact on your credit score. The OP didn't give us enough information. I had assumed all the cards in question were of relatively the same age.
Also, I'm not sure how long it takes for your score to recover after closing an older card. Does it take as long as one of your remaining cards to reach the same age as the one you closed? I wouldn't think it would be quite so bad.
Finally, I'm not sure a credit score drop would have much of an impact in your life anyway, unless you were planning on borrowing a bunch of money in the near future (buying a house, financing a new vehicle, etc.). If you don't plan to borrow money in the next few years, I wouldn't fret about a drop in my credit score. I'm also not convinced it would have a significant impact on things like insurance rates either, but maybe things work differently in the US.
They work differently in the US. Credit score most definitely has an impact on insurance rates.
As for assuming all cards are relatively new - in that case, I agree with you that closing one with an annual fee would be wise.