Yes, of course it depends on term but assume they are equal for the two options.
Will either loan be kept until full term? And, are you sure?
Makes sense. But you are basically discounting at 6% rather than 4%. If I do that with my calculation I get a similar result to you. The difference probably comes from roundng though I did not check.
I had typed part of a response but my baby ate it. But it had started with the assumption that interest rate and discount rate are one and the same. It is common to assume that, but sometimes it leads to odd results like yours where two clearly different loans have the same NPV.
The PV of a series of $600 payments at 4% discount rate = $125,582.60
The PV of a series of $477 payments at 6% discount rate = $79,628.87
You could also discount the payments at the inflation rate. That measures what a dollar tomorrow will be worth, right? In that case the series of payments on the 6% loan is worth $133,517 today; the 4% loan payments are worth $106,318; the difference being $27,199.
So, based on that I might pay between $20,371 and $27,199 in refinancing costs.
I have never heard of a loan with 20+ points, so like you say, it seems like it's always worth paying points. I guess most people either don't have the cash to spare, or plan to move/refinance/etc before up-front points would pay off.