The GTA Online thread
Discussion
RichwiththeS2000 said:
Amazing how these games companies always fail to plan for the right capacity. They even delayed online launch by 2 weeks to make sure they have enough servers.
They plan and sort out servers for player capacity ongoing. There is only going to be a couple days where everybody is trying to get on, after that it will filter out and maintain a steady level - their servers will be set up for this number of people. End of the day they are a business and are not going to invest tens of thousands of pounds into hardware which will be void after a couple weeks.
Plus they can test all day long with 100 players. But switching online on for millions of players globally is much different - So the errors which occured could have been new to them.
I also suspect some of the issues were due to do with PSN, we know thats not the most stable network at the best of times.
Whilst you could say its unacceptable as we are paying customers, if you look at it from their respect its quite reasonable and a compromise they have to make.
Maybe years ago, but these days - not really, seeing as you can get such elastic cloud servers "on demand". A proper infrastructure on their part would be able to scale up and down as demand dictates.
There's no excuse, and it's a shame seeing as they spent so much money developing and perfecting every other element of the game.
There's no excuse, and it's a shame seeing as they spent so much money developing and perfecting every other element of the game.
I suspect the demands being placed on the servers have never actually been experienced on a system in the real world before.
Their server technology to deal with millions of simultaneous gamers is probably untested cutting edge stuff, so there is bound to be teething problems.
Remember more people than ever before (including COD releases) are trying to play at the same time.
Their server technology to deal with millions of simultaneous gamers is probably untested cutting edge stuff, so there is bound to be teething problems.
Remember more people than ever before (including COD releases) are trying to play at the same time.
RichwiththeS2000 said:
Maybe years ago, but these days - not really, seeing as you can get such elastic cloud servers "on demand". A proper infrastructure on their part would be able to scale up and down as demand dictates.
There's no excuse, and it's a shame seeing as they spent so much money developing and perfecting every other element of the game.
THISThere's no excuse, and it's a shame seeing as they spent so much money developing and perfecting every other element of the game.
The financial outlay argument is null and void, and even if this was the case - what price do they put on their reputation?
Couple this with the iFruit for Android (or lack of) issue and they aren't handling this in the best way.
They have spent stacks on marketing, only to foul up in such a dramatic way and destroy a load (though obviously not all) of the goodwill people had for the title.
The online was never going to be perfect as it's impossible to predict the paths such a large number of people will all take once in the main part of the game, but it's quite obvious that when they require you to go through certain paths for the tutorial aspects prior to be letting lose, these will be hit and hit hard - yet issues we are seeing seem to point to these not being covered properly.
I also don't buy the argument that keeps getting rolled out about the numbers of people actually buying the game exceeding predictions and therefore impacting their plans for online - bull, this was always going to be massive and they should have factored for that - they've had 5 years to plan this FFS.
I'm not saying it would stop me buying a R* game, but they just need to get their s
t together a bit more - 6 P's people!Shilvers said:
So, finally managed to do the 'training', but now my car that I fitted a tracker to isn't on the map and I've no number for the insurance co. in my contacts.
Anyone?
did you buy the insurance when you bought the tracker from Los Santos Customs? Your car should appear as the "impound" symbol from Single player mode on the map. Only found this out when some asshat stole my car when i couldn't exit a cutscene from Simeone!!!!! (Another thing to moan about)Anyone?
1st time you do this is free but after that you have to pay
Think we should be very strict with the online crews and boot randoms, had a guy shooting me this morning when I was trying to get him away from the cops, he then kept hunting me down, he had a PHSM tag, but I can't find him in the club, must have a different name and only joined 2/10.
DJKRAZYBOI76 from the USA who are you??
DJKRAZYBOI76 from the USA who are you??
XslaneyX said:
Shilvers said:
So, finally managed to do the 'training', but now my car that I fitted a tracker to isn't on the map and I've no number for the insurance co. in my contacts.
Anyone?
did you buy the insurance when you bought the tracker from Los Santos Customs? Your car should appear as the "impound" symbol from Single player mode on the map. Only found this out when some asshat stole my car when i couldn't exit a cutscene from Simeone!!!!! (Another thing to moan about)Anyone?
1st time you do this is free but after that you have to pay
Anyway. Owning and insuring a car in a game called Grand Theft Auto!! What's the world coming to!?!?
NateWM said:
How long did it take you guys to get into a session for the first race? Mines been saying "Launching Session" for the past ten minutes now.
It seems a bit 'hit and miss' mate.I played last night (xbox) and it took about 2-3 minutes to load to the character design screen, then maybe 5 minutes to actually load me into the game. The first race took about 1-2 minutes to load and then the game seem to smooth out and loading dropped to around 20-30 seconds there after.
Whether you load and how long it takes seems to be down to luck at the minute....
The Beaver King said:
It seems a bit 'hit and miss' mate.
I played last night (xbox) and it took about 2-3 minutes to load to the character design screen, then maybe 5 minutes to actually load me into the game. The first race took about 1-2 minutes to load and then the game seem to smooth out and loading dropped to around 20-30 seconds there after.
Whether you load and how long it takes seems to be down to luck at the minute....
Ok, thanks bud! Sent a invite request to the club just incase any of you can accept it.I played last night (xbox) and it took about 2-3 minutes to load to the character design screen, then maybe 5 minutes to actually load me into the game. The first race took about 1-2 minutes to load and then the game seem to smooth out and loading dropped to around 20-30 seconds there after.
Whether you load and how long it takes seems to be down to luck at the minute....
GT: cossieburn
Have to agree with what others have said re R* dropping the ball on the online. The we got "caught out" by not expecting that many people to try to go online?? Come on pull the other one, this is probably the biggest game release in the last couple of years, surely they had an inkling that it was going to sell by the truckload and most of those people would be trying to get online on day 1?
Also the whole point of cloud computing is that it should give you the ability to be able to rapidly spin up or shutdown server capacity as demand fluctuates, it's the way that many large companies like Amazon etc operate, they will even sell you this service\capacity if you don't want to buy the hardware yourself so surely a company with R* money\clout should have been able to come up with a viable solution to meet the demand and then wind it down after a few days as it settled down?
Lastly you can test this kind of thing long before you've ever gone live these days, you can get testing equipment\load simulators which can literally simulate hundreds of thousands of simultaneous user connections, how else do you think other large companies test their websites\infrastructure before going live?
All in all for a games company who should be on the bleeding edge for this kind of stuff, it's very disappointing that they haven't been able to get this right, no excuses.
Also the whole point of cloud computing is that it should give you the ability to be able to rapidly spin up or shutdown server capacity as demand fluctuates, it's the way that many large companies like Amazon etc operate, they will even sell you this service\capacity if you don't want to buy the hardware yourself so surely a company with R* money\clout should have been able to come up with a viable solution to meet the demand and then wind it down after a few days as it settled down?
Lastly you can test this kind of thing long before you've ever gone live these days, you can get testing equipment\load simulators which can literally simulate hundreds of thousands of simultaneous user connections, how else do you think other large companies test their websites\infrastructure before going live?
All in all for a games company who should be on the bleeding edge for this kind of stuff, it's very disappointing that they haven't been able to get this right, no excuses.
got lucky today going online, saw about 10 people from ph.
not sure if it's because I'm in HK, connection maybe a bit flaky, can't really hear what they were saying on the mic...
joined a mission with...errr...kieranxp5...I think...a mission with a rank of 8, and I joined at rank 3
...died and failed mission about 3 times, then on the 4th time suddenly I was the only one left in the mission! I took it very carefully on my own to finish the mission...took me ages! With only a handgun and running out of bullets, I just managed to kill the last man with 3 bullets left...lol
maybe online is starting to work now...give it a try, but I did go on at about 4pm HK time, that's...err...9am UK I think, might be cause of not many people going online at that time...
not sure if it's because I'm in HK, connection maybe a bit flaky, can't really hear what they were saying on the mic...
joined a mission with...errr...kieranxp5...I think...a mission with a rank of 8, and I joined at rank 3
...died and failed mission about 3 times, then on the 4th time suddenly I was the only one left in the mission! I took it very carefully on my own to finish the mission...took me ages! With only a handgun and running out of bullets, I just managed to kill the last man with 3 bullets left...lolmaybe online is starting to work now...give it a try, but I did go on at about 4pm HK time, that's...err...9am UK I think, might be cause of not many people going online at that time...
Long post!
It's some text I copied to send to some guys in work about server problems
Online video game launches tend to be precarious things. Publishers can buy in vast networks of servers and hire the best software engineers to ensure gamers aren't left out in the cold. But when many thousands hit the infrastructure at once, the results can be wildly unpredictable. We only have to look at the chaotic arrival of EA's Sim City or Blizzard's Diablo III for evidence of this. Both games pretty much collapsed at launch, with fans unable to sign in for many hours and instead jamming the forums with questions and complaints.
So this is going to be an edgy day for Rockstar. Released last month to huge critical acclaim, Grand Theft Auto V has sold in the region of 15m copies, making it the year's biggest entertainment release. And today at noon the publisher is unlocking the ambitious multiplayer mode: Grand Theft Auto Online. Providing fans with a persistent version of the game's world, Los Santos, owners of GTA V can access the new mode for free, join gangs, and get involved in heists, races, death matches and other nefarious activities, all the time earning Reputation Points which unlock new goodies such as weapons and cars.
It's essentially a massively multiplayer game, like World of Warcraft, but set in a modern city rather than a fantasy kingdom. Each time you play, you enter a server that is currently restricted to 16 players, but the entire universe is interconnected – there's even a persistent economy and a stock market that changes dynamically based on what players all over the world spend their in-game cash on.
Of course, with the launch merely hours away the big question is, will it work? Rockstar itself has already admitted that server problems are likely. In a post on its Newswire site last week, the publisher stated:
One thing we are already aware of, and are trying to alleviate as fast as we can, is the unanticipated additional pressure on the servers due to a significantly higher number of players than we were anticipating at this point – we are working around the clock to buy and add more servers, but this increased scale is only going to make the first few days even more temperamental than such things usually are.
Indeed, GTA V owners are already experiencing some of the problems of a connected gaming experience. The game's smartphone app, iFruit, which lets users train lead character Franklin's dog as well as customise cars, has been besieged by server connection delays, as has access to the in-game Snapshot app, used to take and share photos. Some fans are mystified as to why Rockstar failed to anticipate this demand – after all, GTA IV sold more than 25m copies.
But anticipation of huge gamer numbers is one thing, actually preparing for it is quite another. It's not just a case of connecting everyone to one giant online brain. "Although we use the term 'servers' in a lot of cases it's not like there's actually one machine – or a set of machines – that handle the whole job," says Ed Fear, a creative producer at UK studio Mediatonic. "You have machines that do certain functions, or contain certain zones of the world, or hold certain data, and they all operate together to give the impression that you're interfacing with one 'thing'. Unexpected load on one of these areas can unbalance the whole caboodle."
Developers do have a range of methods for load-testing server networks, from running beta programmes (limited numbers of gamers testing the game before release), to designing AI bots that work as virtual testers, to loading the infrastructure with fake work to test the strain.
These are useful tools, but they can only ever provide a limited glimpse at what may happen at launch. "The challenge is scale," says veteran game developer Martin Hollis. "It doesn't matter how much you test with 10 players or with 10,000 players, you can't always exactly anticipate what will happen with 1,000,000 players. Of course you should do your homework. Of course you should test with 1m robots, or more exactly, with automated scripts generated by people experienced in preparing automated scripts for testing massively multiplayer games. You can simulate a lot. But you can't simulate everything, and bitter experience has shown that you can't foresee everything. There is always a surprise or two with every massively multiplayer game, doubly so if you are dealing with millions of players in a spike."
What pre-release testing also fails to account for is the sheer unpredictability of the consumer swarm. In the aftermath of EA's troubled SimCity launch earlier this year, Lucy Bradshaw, studio head at the title's developer Maxis, told Polygon, "A lot more people logged on than we expected. More people played and played in ways we never saw in the beta."
It's the last bit that is most telling. Beta testers tend to understand and work within the confines of unfinished software. They're often fans of the developer or the series, and they feel honoured to be part of the testing process. Paying consumers will react very differently to a slow server or bugs in the system – they'll re-start, they'll log out and log back in – they'll get angry. Furthermore, if they do get on, they will interact with the game in very different ways, testing its boundaries, trying weird stuff – doing things the developers didn't expect. You can't predict what 2m people are going to do.
And you also can't predict the vagaries of several million different internet connections, broadband suppliers, and computer infrasturctures along the pipeline of the internet. "The real key to this is that games require all of the clients (usually 32 to 64) to be perfectly synched," says Andrew Smith of Split Milk Studios. "As soon as one player becomes out of step, the knock-on effects become nightmarishly difficult to contain.
"Picture a ten-pin bowling scenario: the bowler is the server or the game, and the pins are the clients or players. As the ball hits, the first pin falls, and knocks on to the next and it all works. However, if any pin freezes in place for a split second, at random, it completely ruins your ability, as a bowler, to reliably and accurately go for strikes. Now apply that to a game as complex as GTA and you can see why a simple beta is never going to be enough – there are millions of pins."
So the key message from Rockstar is, bear with us. This is effectively a mass open beta. As the website puts it: "The first couple of weeks we expect to be heavily focused on tuning the experience as it goes from internal testing to the reality of being played by tons of people in the real world so that all the usual teething problems for an online game are overcome. We hope it will all run incredibly smoothly, but please bear with us if it doesn't, and help us fix any and all problems!"
In short, if you were planning to take the afternoon off to go online with your friends and rob security trucks, you may want some back up entertainment prepared for when the servers slow, or things go wrong. Because if past experiences of major online game launches tell us anything it's that it won't just be the flash cars inside GTA Online that will be crashing.
____________________________________
____________________________________
"The number one problem is bottlenecks. You can distribute many aspects of your server infrastructure, but if you have a point where the need for data coherency outweighs the desire to parallelise your systems then that is likely to become the weak point in the chain. For example, after a maintenance period, your servers are down, when the systems come back up, everyone will be wanting to login – which then really hammers that part of the system. And distributing login systems becomes harder as they all need access to the same data.
"You can distribute the data based on geographical location, but that then runs the risk of increasing costs whilst not actually matching the levels of demand. There are many ways to approach this, but the fact remains that this is something that is still relatively new to a lot of game developers and trying to do it yourself can often result in a system that works but is very susceptible to collapse under load. Building systems that scale well under load is not a new problem, but it is new for game developers and quite often the budgets in terms of time and staff expertise will be skimped on in favour of focusing on much more visible, whiz-bang features.
"Finding good beta testers is also hard – people generally want to play, not test the game. Professional QA teams will put together detailed test plans, which ensure coverage of all aspects of the game, but executing that at the level required to fully stress your back-end systems would require an unfeasibly large QA department. If you architect your systems well enough, you can build automated tests for this, but humans will introduce a degree of unpredictability that can't really be planned for (and therefore, tested). If you have test teams in Paris, London and Frankfurt, you may find that your network connectivity to say, Rome, has issues with bandwidth and latency that you didn't anticipate.
"Consumer broadband is in a better place now that it was 10 or even five years ago. ISPs are more aware of gaming and they tend to provide routers that will work with services such as Xbox Live and PSN without problems. The good ones will ensure that traffic for those systems is routed and prioritised to give their customers a good experience.
"That said, there are still many factors involved and someone using a Wi-Fi connection in a noisy (electro-magnetic) environment may see spikes of packet loss and/or latency that can have consequences for the quality of the experience they get. In the worst case, it may result in them appearing as having lost their connection to your server. They then have to reconnect which can put further stress on those login servers. A brief outage in a major backbone on the Internet is not unheard of, the effect on Netflix, iPlayer may even be masked by the buffering those systems use. You probably won't notice when using email or the web. But if you're in an online game session, it could cause a few thousand players to lose their connection to the server. They then all have to login again, and the system comes under increased pressure.
"The biggest problems come from spikes in activity and your systems not being able to adequately handle those spikes. Companies that run MMO games have gained a lot of experience in this. Games like Grand Theft Auto started out with much simpler multiplayer expectations, with the game experience being much more transient and isolated. Now, as the amount of persistent data about a player increases and they make progress from one session to the next, this leads to more data storage. The amounts are trivial compared to the storage capacity of modern cloud services, but you then need to have robust systems to store, retrieve and update that data. It needs to be fault-tolerant and above all resistant to corruption or loss. The only thing that will infuriate your player base more than not being able to play the game is losing some or all of their progress."
It's some text I copied to send to some guys in work about server problems
Online video game launches tend to be precarious things. Publishers can buy in vast networks of servers and hire the best software engineers to ensure gamers aren't left out in the cold. But when many thousands hit the infrastructure at once, the results can be wildly unpredictable. We only have to look at the chaotic arrival of EA's Sim City or Blizzard's Diablo III for evidence of this. Both games pretty much collapsed at launch, with fans unable to sign in for many hours and instead jamming the forums with questions and complaints.
So this is going to be an edgy day for Rockstar. Released last month to huge critical acclaim, Grand Theft Auto V has sold in the region of 15m copies, making it the year's biggest entertainment release. And today at noon the publisher is unlocking the ambitious multiplayer mode: Grand Theft Auto Online. Providing fans with a persistent version of the game's world, Los Santos, owners of GTA V can access the new mode for free, join gangs, and get involved in heists, races, death matches and other nefarious activities, all the time earning Reputation Points which unlock new goodies such as weapons and cars.
It's essentially a massively multiplayer game, like World of Warcraft, but set in a modern city rather than a fantasy kingdom. Each time you play, you enter a server that is currently restricted to 16 players, but the entire universe is interconnected – there's even a persistent economy and a stock market that changes dynamically based on what players all over the world spend their in-game cash on.
Of course, with the launch merely hours away the big question is, will it work? Rockstar itself has already admitted that server problems are likely. In a post on its Newswire site last week, the publisher stated:
One thing we are already aware of, and are trying to alleviate as fast as we can, is the unanticipated additional pressure on the servers due to a significantly higher number of players than we were anticipating at this point – we are working around the clock to buy and add more servers, but this increased scale is only going to make the first few days even more temperamental than such things usually are.
Indeed, GTA V owners are already experiencing some of the problems of a connected gaming experience. The game's smartphone app, iFruit, which lets users train lead character Franklin's dog as well as customise cars, has been besieged by server connection delays, as has access to the in-game Snapshot app, used to take and share photos. Some fans are mystified as to why Rockstar failed to anticipate this demand – after all, GTA IV sold more than 25m copies.
But anticipation of huge gamer numbers is one thing, actually preparing for it is quite another. It's not just a case of connecting everyone to one giant online brain. "Although we use the term 'servers' in a lot of cases it's not like there's actually one machine – or a set of machines – that handle the whole job," says Ed Fear, a creative producer at UK studio Mediatonic. "You have machines that do certain functions, or contain certain zones of the world, or hold certain data, and they all operate together to give the impression that you're interfacing with one 'thing'. Unexpected load on one of these areas can unbalance the whole caboodle."
Developers do have a range of methods for load-testing server networks, from running beta programmes (limited numbers of gamers testing the game before release), to designing AI bots that work as virtual testers, to loading the infrastructure with fake work to test the strain.
These are useful tools, but they can only ever provide a limited glimpse at what may happen at launch. "The challenge is scale," says veteran game developer Martin Hollis. "It doesn't matter how much you test with 10 players or with 10,000 players, you can't always exactly anticipate what will happen with 1,000,000 players. Of course you should do your homework. Of course you should test with 1m robots, or more exactly, with automated scripts generated by people experienced in preparing automated scripts for testing massively multiplayer games. You can simulate a lot. But you can't simulate everything, and bitter experience has shown that you can't foresee everything. There is always a surprise or two with every massively multiplayer game, doubly so if you are dealing with millions of players in a spike."
What pre-release testing also fails to account for is the sheer unpredictability of the consumer swarm. In the aftermath of EA's troubled SimCity launch earlier this year, Lucy Bradshaw, studio head at the title's developer Maxis, told Polygon, "A lot more people logged on than we expected. More people played and played in ways we never saw in the beta."
It's the last bit that is most telling. Beta testers tend to understand and work within the confines of unfinished software. They're often fans of the developer or the series, and they feel honoured to be part of the testing process. Paying consumers will react very differently to a slow server or bugs in the system – they'll re-start, they'll log out and log back in – they'll get angry. Furthermore, if they do get on, they will interact with the game in very different ways, testing its boundaries, trying weird stuff – doing things the developers didn't expect. You can't predict what 2m people are going to do.
And you also can't predict the vagaries of several million different internet connections, broadband suppliers, and computer infrasturctures along the pipeline of the internet. "The real key to this is that games require all of the clients (usually 32 to 64) to be perfectly synched," says Andrew Smith of Split Milk Studios. "As soon as one player becomes out of step, the knock-on effects become nightmarishly difficult to contain.
"Picture a ten-pin bowling scenario: the bowler is the server or the game, and the pins are the clients or players. As the ball hits, the first pin falls, and knocks on to the next and it all works. However, if any pin freezes in place for a split second, at random, it completely ruins your ability, as a bowler, to reliably and accurately go for strikes. Now apply that to a game as complex as GTA and you can see why a simple beta is never going to be enough – there are millions of pins."
So the key message from Rockstar is, bear with us. This is effectively a mass open beta. As the website puts it: "The first couple of weeks we expect to be heavily focused on tuning the experience as it goes from internal testing to the reality of being played by tons of people in the real world so that all the usual teething problems for an online game are overcome. We hope it will all run incredibly smoothly, but please bear with us if it doesn't, and help us fix any and all problems!"
In short, if you were planning to take the afternoon off to go online with your friends and rob security trucks, you may want some back up entertainment prepared for when the servers slow, or things go wrong. Because if past experiences of major online game launches tell us anything it's that it won't just be the flash cars inside GTA Online that will be crashing.
____________________________________
____________________________________
"The number one problem is bottlenecks. You can distribute many aspects of your server infrastructure, but if you have a point where the need for data coherency outweighs the desire to parallelise your systems then that is likely to become the weak point in the chain. For example, after a maintenance period, your servers are down, when the systems come back up, everyone will be wanting to login – which then really hammers that part of the system. And distributing login systems becomes harder as they all need access to the same data.
"You can distribute the data based on geographical location, but that then runs the risk of increasing costs whilst not actually matching the levels of demand. There are many ways to approach this, but the fact remains that this is something that is still relatively new to a lot of game developers and trying to do it yourself can often result in a system that works but is very susceptible to collapse under load. Building systems that scale well under load is not a new problem, but it is new for game developers and quite often the budgets in terms of time and staff expertise will be skimped on in favour of focusing on much more visible, whiz-bang features.
"Finding good beta testers is also hard – people generally want to play, not test the game. Professional QA teams will put together detailed test plans, which ensure coverage of all aspects of the game, but executing that at the level required to fully stress your back-end systems would require an unfeasibly large QA department. If you architect your systems well enough, you can build automated tests for this, but humans will introduce a degree of unpredictability that can't really be planned for (and therefore, tested). If you have test teams in Paris, London and Frankfurt, you may find that your network connectivity to say, Rome, has issues with bandwidth and latency that you didn't anticipate.
"Consumer broadband is in a better place now that it was 10 or even five years ago. ISPs are more aware of gaming and they tend to provide routers that will work with services such as Xbox Live and PSN without problems. The good ones will ensure that traffic for those systems is routed and prioritised to give their customers a good experience.
"That said, there are still many factors involved and someone using a Wi-Fi connection in a noisy (electro-magnetic) environment may see spikes of packet loss and/or latency that can have consequences for the quality of the experience they get. In the worst case, it may result in them appearing as having lost their connection to your server. They then have to reconnect which can put further stress on those login servers. A brief outage in a major backbone on the Internet is not unheard of, the effect on Netflix, iPlayer may even be masked by the buffering those systems use. You probably won't notice when using email or the web. But if you're in an online game session, it could cause a few thousand players to lose their connection to the server. They then all have to login again, and the system comes under increased pressure.
"The biggest problems come from spikes in activity and your systems not being able to adequately handle those spikes. Companies that run MMO games have gained a lot of experience in this. Games like Grand Theft Auto started out with much simpler multiplayer expectations, with the game experience being much more transient and isolated. Now, as the amount of persistent data about a player increases and they make progress from one session to the next, this leads to more data storage. The amounts are trivial compared to the storage capacity of modern cloud services, but you then need to have robust systems to store, retrieve and update that data. It needs to be fault-tolerant and above all resistant to corruption or loss. The only thing that will infuriate your player base more than not being able to play the game is losing some or all of their progress."
Gassing Station | Video Games | Top of Page | What's New | My Stuff