I apologize for the down-time over the past few hours. Things seem to be back to normal. Let’s cross our fingers.
The great part about writing this site is that I’m able to help others work toward their financial goals. I love to share what I learn, and to read what you are doing to achieve your dreams. There’s a good community here, and I’m proud of that.
One down-side is that it takes a lot of processing power to run the site. WordPress — the engine behind this blog — is a resource hog. When I get a burst of traffic, as I did this morning, then bad things can happen. I’ve been trying to scrimp by with just this server, but it’s not really doing the job. As a result, I’ll probably order I have ordered a second server to off-load the database. If I understand correctly, this will greatly reduce the load on the site during high-traffic periods.
Anyhow — I’ve spent all morning pulling my hair out about this. Things look fixed now, and I’m working with my host to prevent this sort of thing from happening in the future. But I haven’t managed to get anything written for tomorrow. If there’s another guest post in the morning, don’t be surprised!
I feel for you JD! I’m such a Luddite when it comes to new technology. In fact I just learned the other day that a “plasma screen” is a kind of TV monitor. I always thought it was some type of blood test.
- Jeff Yeager, Ultimate Cheapskate
Take a breath… it’ll be fine!
Your doing an awesome job
wp-cache is your friend
Yeah — look into caching (though I don’t know anything about blogs specifically). But there’s no reason why WP should be hitting the database too often — it’s not like the article is changing by the minute. If this is the case, offloading the database to a separate server would be a waste; the database shouldn’t be working too hard and the webserver will.
James
I have WP-Cache on. I even have SuperCache installed.
According to my host, one problem is the way my server is set up. It’s optimized for reads, but it’s actually doing twice as much writing as reading. They don’t know why. I don’t either, though I wonder if comment spam might not have something to do with it.
They did suggest that one option is to move from a RAID-1 configuration to a SCSI drive. Anyone know how effective that would be?
JD,
Based on that suggestion I suspect they don’t know what they’re talking about. A RAID-1 configuration two hard disks mirrored together for redundancy. SCSI is a type of connection between the hard drive and the system board. SCSI hard disks mirrored together with RAID-1 is a pretty standard server configuration.
There’s no such thing as ‘too much success’! If it becomes a problem, I’ll gladly take some off your hands.
JD,
You have been a great inspiration for me. Keep the good work going!
Remember that the info they gave me is being filtered through the J.D. “limited knowledge about computers” filter before being posted here.
I do know that my current server is configured RAID-1. I do know that the drives aren’t SCSI, and that they suggested that as a solution. I do know that my site is doing twice as many writes (160) as reads (80). (Don’t know units of measure on that, or what it actually means.) I understand that high traffic loads make it so the disk can’t keep up. I know that caching helps, but I already have caching turned on in WP (and they installed additional caching software, too, which was causing the blank pages earlier).
I don’t know much else.
SCSI drives would likely improve read/write performance and I would certainly look into purchasing them (they are a bit more expensive) but there probably other things going on as well. Are the forums hosted on this machine as well?
Yes, the forums are on this machine as well.
At some point, I would think it makes perfect sense to split out the web server from the database server (whether or not now is the time is a good question). This is done for *most* web applications/sites that pull from a DB. If you were to do this, you could focus on maximizing performance on the servers separately based on their function.
Never too much success, but definitely well-deserved.
I admire you.
JD,
SOme suggestions. First of all find out exactly where these reads/writes are coming from.
Secondly, if you can get them them to give you CPU breakdown by process that would tell you which part is being a hog. You might be able to get away with moving your website to a shared server and just running the database off your dedicated server or visa versa.
Thirdly do they offer something like cacti (http://cacti.net/) for monitoring your servers. That would give you nice pretty graphs to help find the bottleneck.
Also if they don’t know why the writes and reads are at the level they are at, perhaps an outside consultant could tweak your configuration.
I hear ya pal. I run two sports sites one with Wordpress, which has become a bear as I’ve been getting tons of hits on there. My other site though I run with joomla from joomla.org, and it is awesome. I’m not promoting it, but between the two I like it a little better because of better pluging, nicer look, and it give a professional feel to my blog (it doesn’t make it look so bloggy).
I love your site especially your article on how you got out of debt…I’m at that point right now after a divorce, and make 6-figures but can’t seem to get it right…I’m hoping your site will help. Good luck keep up the great work and thanks.
Paul G
One possible solution (which requires someone who knows a LOT about computers) is to set up a server on Amazon Web Services “elastic compute cloud” or EC2. It is basically a virtual server running on their system that can be programmed to bring up an 2nd, 3rd, 4th (or more) server to balance the load and handle the traffic spikes.
Just a thought.
Dear J.D.,
first of all: keep blogging
- I’m a daily reader of GRS and I really like the way how you shares your financial knowledge evolution in your own personal way, it helps me a lot to keep on the right track and I’ve learned a lot as well. I’m a software engineer and working on scalability issues and I don’t really know how techie person are you so I don’t want to go into details but if you would like to do all the job alone, I prefer you a good source to get some good tips, it’s: http://highscalability.com/ . If you don’t want or can’t handle the increased traffic alone then I prefer to hire a consultant or - it sounds painful but probably that’s better - to move to a hosted, managed blog service (there are many out there). I hope you will find a great solution.
Anyway since I’ve read your stories and you wrote you will edit this blog in full time, the challenges of your life, risks & problems will also change (as always). I think it’s a great opportunity for you - and for us - to look forward and share the different challenges and experiences - to give a path to us what are the risks and in a life when you’re doing your own way. So show us the next steps.
Good luck!
regards,
Tamas
Every silver lining has a cloud.
I’m not tech-y enough to add any suggestions, but I thought I’d add to your server load anyway. Aren’t I generous
.
I’d recommend getting someone (a friend, or pay someone) to trudge through your logs and optimize the crap out of everything.
You can benefit hugely from a PHP accelerator and optimized database caching settings. You need a pro.
While a secondary server for the database would work fine, that’s not going to LESSEN the load, only make things “feel faster” when the load is that high. What you really need is output compression (for raw traffic), DB caching (to lessen the processing load on the DB machine), and output caching on the PHP side.
Additionally, it might help to offload images to a site like flickr or picassa.
1. For both security and Performance separating the DB and Web server is better.You can tune each server to do a specific thing while not limiting the other tasks it does, for security, it means only the website is available to the internet, the DB is only accessible internally from the website. Im not sure if you own your servers or rent space, but if renting one option is to just rent a bigger server hardware set than virtualize the server software so it will act as 2 independent servers (usually increases performance over having everything on a single server install) This gives you the benefit of separating the servers while saving you the cost of buying a second server and renting more rack space.
2. You have twice the writes as you do reads because it is RAID 1. RAID 1 has 2 drives which mirror each other, so it has to do each “write” twice (one for each drive) it will actually split the reads between the drives (which is why RAID 1 is seen as a performance booster)
3. If they told you to move from RAID 1 to SCSI they were confused. SCSI is a type of drive connector common in servers because they have a history of being very fast (10k RPM and fast transfer) and also highly reliable. RAID is the configuration of the drives in relation to each other. Each RAID configuration has its ups and down (I have a post about it somewhere on my blog)
Additionally, it might help to offload images to a site like flickr or picassa.
Flickr is not an image hosting site, it is a community site for photos. As such, Flickr’s terms of use require that any image hosted on Flickr link back to Flickr. Just FYI.
Additionally, it might help to offload images to a site like flickr or picassa.
Flickr is not an image hosting site, it is a community site for photos. As such, Flickr’s terms of use require that any image hosted on Flickr link back to Flickr. Just FYI.
That and hosting images isn’t resource intensive, it is bandwidth intensive and thats not the problem he is having.
J.D.
Let me quick explain RAID 1. I think it’s necessary, and hasn’t quite been explained yet. Keep in mind this explanation doesn’t have anything specific to web hosting.
RAID 1 makes two complete copies of your data on different disks. When you need to write something to you “hard disk” your “computer” writes it to both separate drives inside your server. When you read something off of your “hard disk” your “computer” reads it off of one hard drive only, (it actually reads half the data off of one and half off of the other to get the data faster.) So in this context, you sort of are writing twice and reading once…
The advantages of RAID one are for data redundancy, meaning you can have one drive explode and still have another with everything you love on it. And secondly, you have increased your read speed.
SCSI drives are typically faster, and more reliable vs. IDE drives. RAID operations may need processing power. So it might make sense to replace an IDE RAID1 with SCSI no-raid. Really though, if you have a hardware RAID configuration, then the CPU consumption is pretty much nothing.
Real servers use RAID 5 or RAID 6 (check out wikipedia) and SCSI drives. I’ve built RAID 5 and RAID 6 machines for commercial purposes, email me if you want more info.
@Jeffeb3 I don’t think it’s fair to say “real servers” use Raid 5 v. RAID 1. they have different intended purposes. RAID 1 has a performance increase over RAID 5 but you give up half your total hard drive space to achieve that. In RAID 5 you only lose a single drive typically out of 3 or 5, and you can go on from there about the other benefits from each, but for his purposes, RAID 1 may be a better solution since he really does need the quick reads.
Switching to SCSI is probably a good bet through. Although SATA has comparable speeds in data transfer, their reliability is lower but they are significantly cheaper in the short term (you may end up spending more replacing the drives over the life of the server)
I gave the go-ahead for the separate database server. Several people have e-mailed to offer help. Thank you. Once I get everything set up, I’ll get the specs for the machines, and make it a priority to learn more about managing them. You’d think as a guy who did some computer consulting for five years, I’d be more up on this. But the truth is I just did simple office networks and know very little about modern web servers.
This is a “good” problem to have all things said. Congrats on your success! And good luck with the capacity upgrades.
Peter
Wow. That’s a problem with success–sometimes too much of it does you (or your server) in.
I can only hope to have that problem!
-
Ryan
http://uncommon-cents.net/
A proper server configuration should be able to handle the load, given decent specs, particularly if the excessive writes situation is identified and dealt with. If it turns out to be some type of temporary write that is running the drives, you could install more memory and use a ramdisk for that particular process. I have some spam scanning software temporarily running on a virtual server under Xen on another sever of mine, it’s using two IDE drives setup as Raid-1, but I installed a single SCSI drive just for temporary mail processing that almost completely smoothed things out. If the situation really is that the server is over-loaded, you could look into a more scalable solution by using multiple lower-spec’d server and load balancing between them with something like Linux Virtual Server.
[...] of my favorite personal finance blogs is Get Rich Slowly; today J.D. talks about the issue of being too popular of a blog! I can only hope for that [...]
I second what Aaron Griffin said about a PHP accelerator, which is really what’s known as an “opcode cache.” PHP scripts are recompiled into machine code on every page load, and the opcode cache sits between the compiler and the execution steps, grabbing the compiled code and executing that with subsequent page loads. Long story short: it makes big PHP scripts like Wordpress really freakin’ fast CPU-wise.
I’m quite partial to APC, the Alternative PHP Cache, but according to your HTTP headers, you’re hosting on IIS, so that would be out of the question. APC makes heavy use of Apache-only features. If you can switch your hosting to Apache and get APC installed, that would be a powerful and pretty stable addition for a PHP-heavy site like yours.
We Live and We Learn
Keep up the good work!
J.D., I think the link on Lifehacker in this case has helped push a lot of people in this direction.
However I’d keep an eye on the forums as well. Having run a few of those in my time I know resource hungry they can be.
Just over 900 members might not be the biggest forum membership in the world, but that along with the posts and any scheduled tasks can help put extra strain on a server.
Not sure if it is a route you want to go down, but you might want to do something with your robots.txt file. While you might not want to stop search engines crawling for content a delay can be helpful in regard to preventing a quick and large burst in traffic.
Ha, I want your problems
Perhaps it might be worth the money to hire a freelancer/pro to check things out. I don’t know what kind of traffic exactly you’re talking about, but it might be cheaper than committing another $200/month every month.