I had a dilemma. I really, really, really wanted to learn how to play the guitar. But wait! There had been a time when I really, really, really wanted to learn kendo. And how to play the recorder. And how to learn Spanish, and Japanese.
Well, I'm a fairly decent recorder player, fluent in Japanese, passable in Spanish, and am absolutely incompetent in kendo. But I was hesitant about getting a new hobby when I wasn't sure that I'd get my money's worth.
Luckily, my sister had a guitar she was willing to let me borrow; I got my guitar for the price of some picks and strings, to test the waters.

If you read books like "The Renaissance Soul" and "Refuse to Choose" and slap yourself on the forehead saying "...That's me!" then budgeting might have its own challenges for you. These books describe a personality type ("scanners" in "Refuse to Choose," "Renaissance souls" in the eponymous book) at the opposite end of the continuum from a person like Mozart who had one single driving passion--music--from the beginning of his life to the end. Scanners (to use the less pretentious term) have a dozen different interests and take up new ones on a whim, but are intense and devoted to their passions... even if for only a short time.
Most budgets have room for one hobby. But twelve hobbies? That gets expensive very quickly. And it can take its toll emotionally when you look at a project that's sat abandoned for months and realize that you didn't get anything out of it: you "wasted" your money. So how do you enjoy life at this end of the continuum without running up your credit cards?

(1) Lose the guilt.
Go ahead and read one of the above-mentioned books if you want to go into detail, because this is just the five-minute version. Scanners get bored easily, and they may be more excited by learning a new skill than by using what they've already mastered. When you look at the things you've abandoned, rather than feeling guilty for wasting your money, thing about what they have given you: new knowledge, new skills, a new challenge. You may not want to get anything more out of them, but that doesn't mean the money was wasted.

(2) Test out the waters.
It's in the nature of my personality that if I want to try something out, I want to dive into it completely. If I'm taking up a new instrument, why start out with a cheap beginner instrument? I'd just have to replace it anyway eventually. And at the moment when I want to buy it, that instrument is my One True Passion (at least for that week).
If you are a scanner, recognize yourself to be a scanner. You do not have to buy $1000 of equipment every time that you have a new hobby you want to experiment with. You can:
-Ask around to see if a friend has equipment you can borrow
-Buy used equipment from Craigslist
-Rent
-Take a class where the equipment is provided
-Buy entry-level equipment rather than the best
-Use the library to check out books on your new hobby

(3) Spend money in proportion to what you actually spend time doing.
You could put $5 in the jar for a new clarinet every time you spend an hour practicing. You could pay yourself 50 cents for every mile you ride on your bike. This isn't about punishing you if you don't ride your bike or play the clarinet; see (1), lose the guilt. You should feel free to take up new whims and pursue them. But when it comes to buying that $1500 racing bike, you'll probably feel better if it doesn't end up sitting in the garage.

(4) Budget for your whims.
Set aside perhaps $40 or $60 a month for your hobbies... and let yourself spend that money with the provision that you won't feel a smidgen of guilt about what you spend it on. This should ideally be separate from the money that you're saving up to buy a new DSLR camera or a kayak; it should be for what you want in the moment (although if you go a month without having something you need to buy, by all means put it in the kayak fund).

(5) Beware of acquisitiveness masquerading as a sudden new passion in something.
The thing about hobbies is that they are almost all associated with really cool stuff. It is normal to lust after a beautiful violin or copper cookware. It's normal to get that mixed up with the desire to play the violin, or be a really good chef. But if you spend more time browsing the Williams-Sonoma catalog than in the kitchen, that might be a sign you're buying into a fantasy rather than a genuine interest.

(6) Not all hobbies are expensive.
It is almost free to be a blogger, a writer, a reader of library books. It is very cheap to sketch, or take up the tin whistle, or work on open source software, or rent classic movies on Netflix... don't steer yourself away from skydiving if it's really what you want to do, but it's nice to have a few hobbies that are very cheap, if only because there's not as much pressure.

In this regard, I think your budget should be independent of how you spend it. If you've budgeted $200/month for your hobby, that doesn't give you permission to spend $400/month for 2 hobbies. You don't have to do the same hobby each month but you have to choose, be responsible, and stay within your budget. If you can't choose, you have to split it up and spend $100/month on 2 hobbies (or however you decide to proportion it). If you can't do that either, you're going to get yourself into trouble of your own making.

I too am a person with serial obsessions, some of which have stayed with me all my life (so maybe they're not so "serial" after all). With big-ticket items like canoes, kayaks, and musical instruments some of the expense can be justified in that you can usually sell these things for not much less than you paid for them if your interests take you elsewhere.

As I get older the overriding constraint is time. Will I have enough time to devote to this [whatever thing I'm going to buy] to justify the expense? I was recently contacted by a flute maker in England about a flute I ordered from him 9 years ago: he is finally starting to work on it. This guy is one of the world's two or three leading makers of wooden simple-system flutes, and he has a long waiting list. It's totally worth waiting 9 years for one of his flutes. But he asked me if I was still interested and I have to say I pondered it for a few minutes. I barely have enough time for the two flutes I already own (each of these flutes is in a different key; I wouldn't own two flutes in the same key but I can justify having three flutes in different keys!). But then I knew I couldn't pass up the opportunity to have this flute even though it'll set me back more than $5,000. I will make the time to play it, and I'm a good enough flute player to justify spending that much on a flute.

Because I teach and perform music on the side, I try to keep it a self-sustaining passion: I try not to spend more on music than I earn from it. But I do make exceptions, and this flute will be one of those.

Wow. I am not alone!

My wife complains that I am interested in everything. One reason that we have so much stuff -- especially books -- is that I've always been keen on one hobby or another. I have scads of stuff related to hobbies that never amounted to more than a passing interest.

When I was in college, I loved astronomy. I bought loads of astronomy books and a nice pair of binoculars. I haven't touched any of this stuff in more than a decade. I once thought that I would like to learn about electricity, so I bought a bunch of manuals. The've never been opened. For a time, I wanted to be a computer programmer. I took this one further than more of my interests. I actually took classes and got two part-time jobs programming. But I hated it. And I'm sure the money I earned barely covered the costs of the books and computers I had purchased.

Sometimes the expenses do pay off. I'm buying a lot of personal finance books lately, but we all know that's been worthwhile. In 1997, I bought a nice bicycle and related gear. I probably spent $1000 all told, but I lost forty pounds and was fit for the only time in my adult life. That was worth it, too.

But most of the stuff I have is just worthless: piano books, harmonica books, information on shortwave radio, a stack of backpacking magazines, etc. Even my photography gear is beginning to gather dust. Basically, I have more interests than I have time.

(It's curious to note that I have the same problem digitally. That is, I have a zillion ideas for blogs, but only enough time to maintain this one and maybe two more. I'd dealy love to start Vintage Pop, but when? How? And Success Daily. And Four Color Comics. And Bibliophilic. And two or three others that I haven't done as much work on.)

In my early 20s I grappled with this problem in my journal, and came to peace with it by concluding that "I'd rather know a little about a lot than a lot about a little." I like that.

I'm constantly telling my boss about things I'm doing or working on on. The other day I was going into detail about some plan I had for something and my other-coworker was standing there giving me the strangest look. My boss turned and told him "His life is one crazy mission after another" I think he summed it up pretty well.

I basically will learn enough about whatever I got into to have a solid understanding of it. After that it either becomes cost prohibitive to continue or I've gone as far as I can without taking classes. At that point I can hold myself in a conversation about whatever it is and I start to los ethe motivation I entered into it with.