Yes, and I've done it - more than once.
The first time I was about to get a promotion I didn't want so I quit and moved to Switzerland and worked as an au pair for a year. It was one of the best experiences of my life. I was 27 at the time and most people though I was insane to quit Vanguard to be a nanny.
The second time I was an operations consultant for a small group in DC. I worked on a large contract at the US Capitol for several years before I couldn't take it any more. I didn't believe in what I was doing, they weren't allowing me to use my skills and they were forcing me to tread water - none of which works well for me. I stayed there as long as I could (until I was at the point where I was literally sick each morning just at the thought of goinig to work) and then I finally snapped and quit. I used that opportunity to start my current company.
The third time was less a matter of leaving my job than leaving my location. But, since I am self employed and my entire business needs to be re-built in a new location it's similar. The reasoning was the same as experience number 2. I was just no longer happy with where I was living and it was impacting my health. I made a promise to myself that I would never force myself to stay in a job that was impacting my health so it was time to leave.
I'm a big believer that you should actually enjoy what you do (shocker!) and that will make up for any salary shortcomings. Of course, this only works if you're ok with not living an extravagant lifestyle. Personally, it is only me and I can live on very little so it makes it easier.