Load testing, stress testing, capacity testing are different types of Performance testing and these all come under Performance testing. Effectively, it is to appraise any user experience in realistic scenarios on your target application. It will allow you to assess machine and infrastructure capability. Performance testing, load, stress, capacity are all based on business requirements. It’s a professional approach to evaluating the performance of an application. It requires the simulation of real load scenarios running against your target applications or websites.
So that means concurrent users.
Now, concurrent users are only a part of the performance test. It’s an important part but really an equally important part is having a mix of relative transactions.
For example, if you just have 10,000 users hitting your homepage, you’re not testing the web application. All you’re testing is the web server. In the case of a content management server, then yeah, then it’s going to hit the backend database and that might be all you need to do. It might be a very simple sales website or something like that. But for testing sophisticated web applications, then you really do need a mix of relevant transactions going through correct use cases with test data running at correct pacing. That’s the speed, which each of the users is running at, and accurately and correctly generated load.
The load must be generated conforming to HTTP standards. If you’re following all of those points, then you end up with realistic load, which will then give you a fingerprint of the response time that the application will deliver. In fact it will give you the response time that you can expect and also the transaction rate that we know the application can deal with at its peak.
This plus maximum supported number of concurrent users if we’re doing a stress test. Now this does require the use of tools and there are a lot of tools out there. There’s a whole commercial and technical spectrum. There’s all the way through the most expensive commercial tools down to what people might think of as free software or open source software. There is still a cost, which has to be evaluated to getting these tools setup, which leads us on to a technical spectrum. You can go all the way down to purely scripted based approach, no GUI at all, to something that is really over simplified and gives you no control over visibility over the load that you’re generating. There’s also software as a service, which is certainly suitable for public web applications and really only simple transactions. So if you’ve got a complex web app, quite often these SaaS vendors are not able to work with you.
The thing is, it has to be realistic.
Really I would say, do it properly or just don’t do it at all because if you do it incorrectly and use the incorrect methods and tools, you can end up with a performance false negative or a performance false positive. A false negative for example would be if you just take any old tool and just record some scenarios without really understanding what you’re doing, fire it at maximum rate using all the CPU power that you have; and the tool says, hey look I brought the web app down, the performance is terrible, the response times are awful! The thing is that test probably wasn’t realistic and more than likely went way over and beyond the performance requirements from the business. So don’t waste time on trying to fix an application that’s actually not broken, it was just reported as broken – it was a false positive.
On the other hand, a tool or service might come back and say, well, this application’s performing fine. In fact, we had a situation with the bank where they had a very well understood set of business performance requirements, it was actually a call center application. Users would come in at 9:00 and work till 5:00 and work all day, there were 1,000 of these users. So they replicated those scenarios and used a tool to do that. The tool came back with very positive results and the application seemed to perform. When the application went live in fact only 250 of the users could actually get it to work. This was down to the fact that the application couldn’t deal with more than 250 users of real load and the fact that the tool had not generated the load correctly and accurately. This is back in the day of HTTP 1.0 and the number of connected threads per user was not accurate in the tool. In fact the load that was generated was well below the equivalent footprint that real users would have generated.
Next Stress, Load, Soak, Spike, Performance testing