Morydd, I used to feel the same way you do. I have a lot of friends who are in the same boat. Occasionally somebody will try to listen to the book instead of read it, and often they find it unpleasant. "I can't concentrate," they say. "My mind wanders." I'm fortunate that I can concentrate on the book when I listen. My mind rarely wanders.
You're right, though, that it's much easier to listen to audiobooks while driving. My daily commute is half an hour each way. I often have to make hour-plus trips into Portland during the day, too, which gives me more time to listen. I used to figure that I could finish a 12-14 hour novel (which is about 300 pages) in two weeks of driving. A longer novel, like Jonathan Strange clocks in at around 30 hours and might take a month to finish.
The key with audiobooks is to find the good ones. That's part of the reason I asked the question. A poorly read book is tedious. I listened to a version of 20,000 Leagues Under the Sea that bored me to tears. But a well-read book is a wonderful thing.
One last point: I find that my comprehension and retention actually increases when I audit a book. For some reason, I'm much better at remembering things that I hear than those that I read.