BUT, I would also like to know if Vanguard automatically stops you if you try to invest too much?
They track how much you contribute and won't let you enter an amount greater than available. They also have an easy "maximum" option that pulls in the max contribution over your desired periods. Of course, this only works if 100% of your yearly IRA contribution is at Vanguard. If you also open an E-Trade account and put $2K in, E-trade has no idea you already maxed out your IRA at Vanguard.
AND for my situations specifically, if I tell Vanguard to automatically reinvest my dividends and capital gains from the index fund I have in my IRA, does that count against my limit? (Basically, I have already invested $4,000 for this year. Do I need to tell Vanguard NOT to automatically reivest dividends and capital gains?)
Reinvestment of dividends and distributions are not contributions.
Perhaps it might be better to use E-Trade as an example. You make a 4K contribution to your brokerage account. It goes into the cash portion. You can then buy stocks/etfs/funds with that money -- and sell at anytime without penalty. Sales/dividends/distributions just go back to the cash portion of the IRA account. It's not a true withdrawal unless money actually transfers out into your bank accounts. Likewise, a contribution only happens when money comes in from your external bank accounts.