Like many people, I have waaaaay too many clothes and don't wear a lot of them. Part of this is that I just like clothes too much, but part of it is that I'm very hard to fit. I'd buy something that didn't quite fit, often because I needed it and couldn't find anything better, and then end up not wearing it much because it didn't fit quite right. Then I'd buy another something that didn't fit quite right in a different way. Or I'd buy something that looked ok at first, but after a day's wear didn't. The things that I bought that did turn out to fit right would get worn to death, but I wouldn't even know why.
I've become interested in sewing and part of that is learning how to fit patterns. I was stunned to realize that most of the clothes I own simply don't fit right and never did. The best thing about learning to sew has been learning how to fit ready-to-wear. (Note that unless you're a size/shape that is really hard to fit in ready-to-wear, you won't save a dime sewing for yourself. Alterations are cost effective, but sewing garments is an expensive hobby. There are tariffs on textiles, but not on clothes made in sweatshops.)
When I first discovered this, I bought some fairly expensive clothes, but they fit. The difference in comfort and look was amazing. Once I got used to how a correct fit looked and felt, I could go back to the consignment shops and thrift stores. Now I buy a lot fewer clothes that I like better and are more comfortable. They look more professional. It's also easier to say "no" to not-quite-right-but-cheap clothes, because I know what's wrong with them.
Everyone's individual fit is different, and learning how to see if something fits takes practice. Some general points: Learn to tell if it fits in the shoulders, because that's easy to get wrong and kind of weird to figure out if it's right or not -- shoulders are often "off." There's no one rule of thumb because there are lots of different styles for garment shoulders, and each has its own way to look right. They're also very difficult to have altered to be right. Make sure the lengths (sleeves, pants, hems, etc.) are good and the neckline/collar doesn't gap -- those are easier to fix. I learned fitting from pattern alteration books, but I'll bet the nice salespeople at expensive clothing stores can help. When you get more advanced, you can look for where darts point, the pitch of sleeves in suit jackets, and other minutia.
The first step for any woman is to make sure your bra fits. If your bra doesn't fit, your shirts can't, and most women are wearing bras that don't fit (which are also varying degrees of uncomfortable). If your bra hurts at all, it doesn't fit. In my experience, Nordstrom has awesome fitters and a good selection of clearance bras. Victoria's Secret, Macy's, and other places simply aren't as good, because they don't have the size range that Nordstrom does -- it's hard to figure out your size if it's not in the store. The measuring system you're supposed to use to figure out bra fit is wrong for most people, giving you too high a band size and too low a cup size (a direct route to shoulder pain). It does work for some people, but not many; I custom-make bras, so I'm not just being contrary here.
I don't have equivalent advice for men, but I have a lot of male friends who hate wearing suits (and often consequently underdress for their jobs). When I see them dressed up, they're invariably in a suit that doesn't fit -- of course they hate dressing up!