Scaling up a web site
Discussion
What are the implications if a web site goes from having a few hundred hits a week to several thousand a day?
I have a site which I developed as a research tool with samples of about 2000 gathered over a month or six weeks. It uses Flash and Cold Fusion (whatever that means)to meausure response times to key questions and gathers quite a bit of information, drops it into Access and from there into Excel etc. Until now I have only run one or two versions at a time, each one bespoke and separate.
This summer I will be launching two dozen new versions, and a user defined version, the use of which I hope to sell or licence to research organisations and charge them a base fee and/or a fee for each return made. The new version will draw content from a database of materials depending upon what the client wants. I will be unable to control the numbers of people they decide to ask to go to the site and there will be numerous differnt versions being generated at any one time.
My web guys have been doing my small sites for a few years and as expected tell me anything is possible! I just wanted some idea of the technical implications should things take off and numbers go from 2000 every month to 2000 a week or 2000 a day.
Is it just a case of buying in more bandwidth?
Are there certain types of coding that are not suitable for large volumes of traffic? The web guys are already saying we need to move away from using Access for the database because the new version is more complex ( it varies content according to the link we provide).
Ummm basically, you can purchse more bandwidth - to a point. Initially no smaller sites, that's your limiting factor (down to cost mostly).
Then you need to start talking to a host who can handle your projected number of hits. Technology wise, you also need to move from Excel\access data sources to something more extensable & robust - ie SQL\MYSQL. The coding will have to change to take advantage of load ballancing (ie spreading the load over 2 or more physical hosts) & clustering (splitting the load between 2 or more hosts) - all of these technologies will provide more uptime & more 'power' to manage the hits. Most of these questions will be answered by the provider if you chose not to host your own servers because they will already have a capability in place which you will leverage for your site. Hosting for your self at this stage probably isn't recommended unless you have a real need to (there are a whole lot of extra considerations there)
the above is just a basic overview, hope it's useful.
Cheers,
A
Then you need to start talking to a host who can handle your projected number of hits. Technology wise, you also need to move from Excel\access data sources to something more extensable & robust - ie SQL\MYSQL. The coding will have to change to take advantage of load ballancing (ie spreading the load over 2 or more physical hosts) & clustering (splitting the load between 2 or more hosts) - all of these technologies will provide more uptime & more 'power' to manage the hits. Most of these questions will be answered by the provider if you chose not to host your own servers because they will already have a capability in place which you will leverage for your site. Hosting for your self at this stage probably isn't recommended unless you have a real need to (there are a whole lot of extra considerations there)
the above is just a basic overview, hope it's useful.
Cheers,
A
You cannot take a site designed for a few hits and just throw processing power at it so it will deal with thousands of people simultaeneously accessing it.
You design a site for scalability from the ground up. Makes it loads more expensive to start as you start out with big-boys-toys like SQL Server - but you only need to buy server grunt afterwards. If your site is based on desktop tools like Access and Excel it will, as fade2grey, says need to be re-written to use a proper, scalable database like SQL Server, Oracle, mySQL or whatever...
That may be simple - depending. It may not be. It may as bad as a total re-write of the site (admittedly with a brilliant specification - one that works already!).
Answering your question in more detail than the above would require a detailed analysis of your site specifically. Which would cost money. Pay your guys to do that and get you a quote. Be as specific as you can about how scalable you want the result to be...
You design a site for scalability from the ground up. Makes it loads more expensive to start as you start out with big-boys-toys like SQL Server - but you only need to buy server grunt afterwards. If your site is based on desktop tools like Access and Excel it will, as fade2grey, says need to be re-written to use a proper, scalable database like SQL Server, Oracle, mySQL or whatever...
That may be simple - depending. It may not be. It may as bad as a total re-write of the site (admittedly with a brilliant specification - one that works already!).
Answering your question in more detail than the above would require a detailed analysis of your site specifically. Which would cost money. Pay your guys to do that and get you a quote. Be as specific as you can about how scalable you want the result to be...
fade2grey said:
Ummm basically, you can purchse more bandwidth - to a point. Initially no smaller sites, that's your limiting factor (down to cost mostly).
Then you need to start talking to a host who can handle your projected number of hits. Technology wise, you also need to move from Excel\access data sources to something more extensable & robust - ie SQL\MYSQL. The coding will have to change to take advantage of load ballancing (ie spreading the load over 2 or more physical hosts) & clustering (splitting the load between 2 or more hosts) - all of these technologies will provide more uptime & more 'power' to manage the hits. Most of these questions will be answered by the provider if you chose not to host your own servers because they will already have a capability in place which you will leverage for your site. Hosting for your self at this stage probably isn't recommended unless you have a real need to (there are a whole lot of extra considerations there)
the above is just a basic overview, hope it's useful.
Cheers,
A
That is great and a good based to have a chat with my web guys. I may sk them to tell me how it would reaxct if traffic went from 2000 a month, to 10000, to 20000 and then upwards to see what changes that would mean and if it could be done onthe hof or if we would lose service.Then you need to start talking to a host who can handle your projected number of hits. Technology wise, you also need to move from Excel\access data sources to something more extensable & robust - ie SQL\MYSQL. The coding will have to change to take advantage of load ballancing (ie spreading the load over 2 or more physical hosts) & clustering (splitting the load between 2 or more hosts) - all of these technologies will provide more uptime & more 'power' to manage the hits. Most of these questions will be answered by the provider if you chose not to host your own servers because they will already have a capability in place which you will leverage for your site. Hosting for your self at this stage probably isn't recommended unless you have a real need to (there are a whole lot of extra considerations there)
the above is just a basic overview, hope it's useful.
Cheers,
A
If you're anticipating under 500,000 visits and 100GB a month, don't worry about load balancing yet, as it's a real pain, particularly with file replication, etc. Just have a decent spec server with 2M bandwidth and good redundancy.
Ditch Access/Excel ASAP. SQL Server is your friend, as much of your existing code for queries will work straight away.
When you're approaching higher figures than the above, and need more complex functionality for reporting with online analysis on the questionnaires, then PM me
Ditch Access/Excel ASAP. SQL Server is your friend, as much of your existing code for queries will work straight away.
When you're approaching higher figures than the above, and need more complex functionality for reporting with online analysis on the questionnaires, then PM me
Agreed with the load balancing comment. Only 'enterprise' sites should bother with that (> 2000 concurrent users). Check out SQL Express edition, its free but does have a size limit so do some maths and work out if it is suitable. Converting from access to sql should be a piece of piss so don't let your developers charge you a fortune.
Good luck.
Good luck.
Forgot to say... download and run this on your testing system to give you an idea of how scalable your system is.
http://www.microsoft.com/downloads/details.aspx?Fa...
http://www.microsoft.com/downloads/details.aspx?Fa...
ukvoyager.info said:
Agreed with the load balancing comment. Only 'enterprise' sites should bother with that (> 2000 concurrent users). Check out SQL Express edition, its free but does have a size limit so do some maths and work out if it is suitable. Converting from access to sql should be a piece of piss so don't let your developers charge you a fortune.
Good luck.
Load balancing is less about # of concurrents and more about availability requirements..Good luck.
We have clients who LB some of their internal systems which maybe only have < 50 people online simul ever, but need it to be online 100% of the time.
Metrics aren't always decisive factors when designing systems, uptime and availability requirements tend to be more important.
JamieBeeston said:
Load balancing is less about # of concurrents and more about availability requirements..
We have clients who LB some of their internal systems which maybe only have < 50 people online simul ever, but need it to be online 100% of the time.
Metrics aren't always decisive factors when designing systems, uptime and availability requirements tend to be more important.
Also it depends on your application. I built a two layer web farm for a heavy internal web application where they've averaged just 30 users per server. The app was badly written and they had no time for a re-design, so we just threw some more hardware at it. I also built a web farm for a high street bank to host brochureware and they're now using just 4 boxes to serve a million page views a day. They also had massive peaks at lunchtime, but the servers coped with it fine. It all depends on what your app is doing.We have clients who LB some of their internal systems which maybe only have < 50 people online simul ever, but need it to be online 100% of the time.
Metrics aren't always decisive factors when designing systems, uptime and availability requirements tend to be more important.
Don said:
You cannot take a site designed for a few hits and just throw processing power at it so it will deal with thousands of people simultaeneously accessing it.
You design a site for scalability from the ground up. Makes it loads more expensive to start as you start out with big-boys-toys like SQL Server - but you only need to buy server grunt afterwards. If your site is based on desktop tools like Access and Excel it will, as fade2grey, says need to be re-written to use a proper, scalable database like SQL Server, Oracle, mySQL or whatever...
That may be simple - depending. It may not be. It may as bad as a total re-write of the site (admittedly with a brilliant specification - one that works already!).
Answering your question in more detail than the above would require a detailed analysis of your site specifically. Which would cost money. Pay your guys to do that and get you a quote. Be as specific as you can about how scalable you want the result to be...
Agree with everything Don has said. None of us are in any position to make assumptions or give advice as to what your new system should look like.You design a site for scalability from the ground up. Makes it loads more expensive to start as you start out with big-boys-toys like SQL Server - but you only need to buy server grunt afterwards. If your site is based on desktop tools like Access and Excel it will, as fade2grey, says need to be re-written to use a proper, scalable database like SQL Server, Oracle, mySQL or whatever...
That may be simple - depending. It may not be. It may as bad as a total re-write of the site (admittedly with a brilliant specification - one that works already!).
Answering your question in more detail than the above would require a detailed analysis of your site specifically. Which would cost money. Pay your guys to do that and get you a quote. Be as specific as you can about how scalable you want the result to be...
If you aren't entirely confident about what your current web developers are telling you, go to another software/web development firm, outline your needs to them and listen to what they have to say. If it's vaguely similar to your first lot, then they're probably both talking sense. If there is a significant difference ...
JamieBeeston said:
Load balancing is less about # of concurrents and more about availability requirements..
We have clients who LB some of their internal systems which maybe only have < 50 people online simul ever, but need it to be online 100% of the time.
Metrics aren't always decisive factors when designing systems, uptime and availability requirements tend to be more important.
Good point. The OP made no mention of uptime so I assumed it was purely a loading question. We have clients who LB some of their internal systems which maybe only have < 50 people online simul ever, but need it to be online 100% of the time.
Metrics aren't always decisive factors when designing systems, uptime and availability requirements tend to be more important.
If I were the opening poster, I would run up the site in the test environment, hammer it to death and see where I was. Then I could get quotes for hardware and software based solutions and make my decision based on short and long term objectives!
Many thanks for that although most of it is a bit too complex for me to understand. I think the reality is that we won't get more than 50-100 concurrent users but will get peaks when invitations are emailed out (in the past we have staggered invitations going out or sent them at night/weekends to spread the demand). The site is quite demanding in that they will each spend 5-10 minutes there completing various tasks timed through flash. It downloads content from the server dependant upon which version we have invited them to complete and then it periodically sends data back to the server.
Gassing Station | Business | Top of Page | What's New | My Stuff