I actually manage django apps on many accounts on Web Faction's web 3 through web 15. I started using Web Faction about a year and a half ago and it's gone down hill so much since then, service response times are totaly unacceptable, we get out of memory warnings when we're using very little memory, well within our limits, support tickets take forever to get answered and most of the new staff is incompetent. I miss the old days. Alas we're moving to a dedicated machine because it's not worth it to deal with the never ending web faction problems.
Noah
· 2 years ago
We're moving from Web Faction to Rackspace.
rezzrovvv
· 2 years ago
I was one who emailed webfaction about your site ;) You were killing my site.
pkenjora
· 2 years ago
Yeah sorry about that, I woke up one day and Arkayne was getting hammered. I tried getting webfaction to upgrade the server but it was all too slow. The site was growing faster than I could optimize.
To be fair, those guys have way too many sites crammed into one shared host. Jeff Croft's site is on the same server as well as a few other popular ones. I think the box we were on was destined to swell beyond capacity sooner or later.
Good and bad news is Arkayne swelled sooner. Sorry for crowding the CPU, should all be fine now since I relocated. Unless one of the other sites does the same, which appearantly as everyone tells me is a nice problem to have. I always thought too much money was a nice problem, not increasing costs.
Mike
· 2 years ago
Anonymous: Are you sure we're talking about the same company? I have had a complete opposite experience with them: fast server and helpful support. I can't speak for all the servers they have but I have 3 accounts on different servers and all of them have a load around 0.5, which is very good. Maybe you got unlucky and your server was more loaded ...
Observer
· 2 years ago
Umm. I handle a *lot* more than 10k hits on a shared webfaction account with Django, and my setup doesn't use much CPU at all. And the sites are fast as heck. Any special reason you don't mention caching at all? Using Django without any kind of caching is a bad idea, no matter what setup you're using. Using Django without any kind of caching on a shared setup should be punishable by law.
Ian Ozsvald
· 2 years ago
I'd like to give a thumbs-up to WebFaction too! We've hosted http://ShowMeDo.com there for a year (and we eat cpu/memory with increasingly regularity now) - Remi, David and co have been incredibly responsive and helpful. I've only had memory-warnings when indeed I've gone over-quota, they're helpful and have offered advice. Switching to memcached was a big win for us, as was optimising SQLObject queries. I'm a little shocked by your comments about the team up there - that's the first bad news I've heard about them...I'd suggest other readers do some research (maybe I've just missed other bad write-ups?) but all I've heard are good things. My own experiences have been great - supportive, responsive and understanding. Ian.
Pete
· 2 years ago
I'm thinking your code must have been grossly inefficient because I handle about 200K hits per day at WebFaction with no pb at all... My code is a basic Django site with a few queries per page. I didn't even have to enable caching, but I did have to fix a few very expensive queries.
pkenjora
· 2 years ago
To clarify, I'm not slamming WebFaction. They have been great and I'm still hosting some of my sites on their machines.
As far as efficiency. I was running a few expensive batch jobs every so often but without those it was a simple query. Fetch a set of indexed records and return them. Nothing too complex, query itself took very little time. My processes hovered arounf 18MB a piece but my CPU tended to spike to around 50%-90% on a hit.
Every site is different I guess. As far as being lucky or unlucky, Web Faction offered to move me to another machine because that one was heavily loaded. I didn't take them up on it because I was worried I'd have the same problem all too soon. From looking around the processes I could tell lots of other popular sites were on there too. And we definately had more than 2 sites on the server.
On the dedicated machine performance is significantly better without any code or query optimization. All things aside, given time and money its the best optimization I could have done at the time.
Tell me more about Django caching, I may be doing it but know it by a different name. What is it and how do I activate it? What are the benefits?
Brian
· 2 years ago
So, was it actually the number of hits per day on your server that was consuming all the system resources, or some background processes running? If it was the background process, how much of the database does it hit, is it a small enough piece that memcache would solve your problem? I can't imagine that 10k hits per day would be that hard on a server, unless each hit results in an overly expensive query....in which case...that should probably be fixed.
It is also worth mentioning, in order to make serverpronto cost effective, you need to know how to sysadmin your site. They won't help you...unless you want to pay them, which blows the whole cheap thing out of the water.I recommend running in vmware server in order to be able to easily move from one serverpronto machine to another..upgrading as necessary. Purchase a second ip address - a 1 time fee of $10 and use bridged networking. Setup iptables on both host and vm....i tossed all that web based admin bullshit it comes with. weak sauce.
Pete
· 2 years ago
I agree about having to maintain your own server: it's a pain in the neck. At first everything runs smoothly but when you start having to apply patches yourself and re-compiling things that aren't in yum you start pulling your hair.
Also, I'd be curious to know how far the 256MB of memory get you. Once you run the operating system, the web server and database server that doesn't leave much left for your actual application and memcache.
With most shared hosting providers, the memory that they give you is for your actual application and you don't have to worry about the memory used by the OS, the web server and database server.
Simeon
· 2 years ago
Re Django caching:
The invaluable James Bennett had a good article on performance tips that included a section on caching (http://www.b-list.org/weblog/2007/nov/27/perfor...). Basically though Django gives you a bunch of options ranging from whole-site caching (eg: any page w/o POST or GET parameters is cached the first time it is generated and reused until the (configurable) time limit is reached.)
Say for example the front page of your website is pulling dynamic data from the DB - the last ten somethings, a menu structure, a content block, etc. The process for generating all that involves a dozen queries, view code that builds data structures and templates that render the data. The result is just a text file though - and if you cache it for 10 seconds and get 100 hits in those 10 seconds only the first hit goes through the whole process of querying the db, rendering data, etc, and the other 99 simply feed the saved file. You can see that this would have a dramatic effect on performance.
If you just have too much dynamic data for a whole site cache you can also cache individual views (as simple as adding a decorator to the view function) or even use caching template tags to cache fragments in a template. There's also access to the Cache API in python code so you can cache objects (like query results, etc). As usual the docs are pretty good...
pkenjora
· 2 years ago
Unfortunately the majority of content pulled by Arkayne is dynamic and unique for each call. I have Apache serving all the static content. The caching approach does sound applicable to a few of my other sites though. I'll give it a try.
As far as running VMWare on Server Pronto, I'm not sure I have the need. VMWare is absolutely awesome by the way. My experience with deploying a django application on Fedora has been great. I used "yum" to install django, mysql, and a fe python libraries. Beyond that setting up Apache and copying my Django application was cake. All in all the setup took a few hours, and I could probably repeat it in 30 minutes this time around. Setting up extra IP addresses and configuring VMWare on the server sounds like a comprable task.
If I ever plan to deploy to multiple servers then VMware may be the right solution.
All Comments
To be fair, those guys have way too many sites crammed into one shared host. Jeff Croft's site is on the same server as well as a few other popular ones. I think the box we were on was destined to swell beyond capacity sooner or later.
Good and bad news is Arkayne swelled sooner. Sorry for crowding the CPU, should all be fine now since I relocated. Unless one of the other sites does the same, which appearantly as everyone tells me is a nice problem to have. I always thought too much money was a nice problem, not increasing costs.
We've hosted http://ShowMeDo.com there for a year (and we eat cpu/memory with increasingly regularity now) - Remi, David and co have been incredibly responsive and helpful.
I've only had memory-warnings when indeed I've gone over-quota, they're helpful and have offered advice. Switching to memcached was a big win for us, as was optimising SQLObject queries.
I'm a little shocked by your comments about the team up there - that's the first bad news I've heard about them...I'd suggest other readers do some research (maybe I've just missed other bad write-ups?) but all I've heard are good things. My own experiences have been great - supportive, responsive and understanding.
Ian.
As far as efficiency. I was running a few expensive batch jobs every so often but without those it was a simple query. Fetch a set of indexed records and return them. Nothing too complex, query itself took very little time. My processes hovered arounf 18MB a piece but my CPU tended to spike to around 50%-90% on a hit.
Every site is different I guess. As far as being lucky or unlucky, Web Faction offered to move me to another machine because that one was heavily loaded. I didn't take them up on it because I was worried I'd have the same problem all too soon. From looking around the processes I could tell lots of other popular sites were on there too. And we definately had more than 2 sites on the server.
On the dedicated machine performance is significantly better without any code or query optimization. All things aside, given time and money its the best optimization I could have done at the time.
Tell me more about Django caching, I may be doing it but know it by a different name. What is it and how do I activate it? What are the benefits?
It is also worth mentioning, in order to make serverpronto cost effective, you need to know how to sysadmin your site. They won't help you...unless you want to pay them, which blows the whole cheap thing out of the water.I recommend running in vmware server in order to be able to easily move from one serverpronto machine to another..upgrading as necessary. Purchase a second ip address - a 1 time fee of $10 and use bridged networking. Setup iptables on both host and vm....i tossed all that web based admin bullshit it comes with. weak sauce.
Also, I'd be curious to know how far the 256MB of memory get you. Once you run the operating system, the web server and database server that doesn't leave much left for your actual application and memcache.
With most shared hosting providers, the memory that they give you is for your actual application and you don't have to worry about the memory used by the OS, the web server and database server.
The invaluable James Bennett had a good article on performance tips that included a section on caching (http://www.b-list.org/weblog/2007/nov/27/perfor...). Basically though Django gives you a bunch of options ranging from whole-site caching (eg: any page w/o POST or GET parameters is cached the first time it is generated and reused until the (configurable) time limit is reached.)
Say for example the front page of your website is pulling dynamic data from the DB - the last ten somethings, a menu structure, a content block, etc. The process for generating all that involves a dozen queries, view code that builds data structures and templates that render the data. The result is just a text file though - and if you cache it for 10 seconds and get 100 hits in those 10 seconds only the first hit goes through the whole process of querying the db, rendering data, etc, and the other 99 simply feed the saved file. You can see that this would have a dramatic effect on performance.
If you just have too much dynamic data for a whole site cache you can also cache individual views (as simple as adding a decorator to the view function) or even use caching template tags to cache fragments in a template. There's also access to the Cache API in python code so you can cache objects (like query results, etc). As usual the docs are pretty good...
As far as running VMWare on Server Pronto, I'm not sure I have the need. VMWare is absolutely awesome by the way. My experience with deploying a django application on Fedora has been great. I used "yum" to install django, mysql, and a fe python libraries. Beyond that setting up Apache and copying my Django application was cake. All in all the setup took a few hours, and I could probably repeat it in 30 minutes this time around. Setting up extra IP addresses and configuring VMWare on the server sounds like a comprable task.
If I ever plan to deploy to multiple servers then VMware may be the right solution.