Level 17. Load Testing

More specifically, the following types of load testing are distinguished:

1. Performance testing – measures the speed of the system under ideal conditions and maximum load.

2. Load testing – the same performance tests, but with the system subjected to various loads.

3. Volume testing – the application is loaded with a large amount of data to determine when the conditions are reached under which the system stops working.

4. Stress testing – the behavior of the system when resources run short (CPU resources, disk space, network outages, and so on). Stress testing lets you check how well the application and the system as a whole hold up under stress, and also assess the system's ability to recover, that is, to return to a normal state after the stress is removed. In this context, stress can mean raising the intensity of operations to very high values or an emergency change to the server configuration.

5. Stability or reliability testing (Stability / Reliability Testing).

The goal of stability (reliability) testing is to verify that the application keeps working during prolonged operation (24 hours or more) under an average load. Operation times play a secondary role in this type of testing; stability takes first place: no memory leaks, no server restarts under load, and other aspects that affect stability specifically. In stability testing, the duration is dictated only by business logic, and it can last a week, a month, or longer.

1. Testing under the working load

2. Testing under increased load

3. Peak load testing

4. Testing with large volumes of data

5. "Dying gracefully" testing

6. Tuning and playing with settings

7. "Hardware" questions

8. Assessing system survivability

9. Recommendations for the patient

At the design stage.

• Which system architecture is better: three-tier or four-tier? Where and how should files be stored? Will the chosen language and database deliver the performance we need? And so on.

At the programming stage.

• Which database schema is better? Should we give up DB normalization in favor of performance, or the other way around? How should the GUI be built? What is better: triggers or foreign keys?
• Which part of the code should be optimized first?

At the testing stage:

• The maximum performance of the system.
• Determining how much the response time and operation duration grow as the load increases.
• Determining the limit of the program's applicability in terms of the number of users.
• Determining how the system configuration affects performance.
• The peak load on the system.

At the delivery stage:

• Do the architecture and network configuration meet the performance requirements?

A classic mistake is to run performance testing only at the testing stage.

In this case, you'll have to fiddle with the settings and with processing the test results. Our actions:

• Think about how to install the version and where to trim or cut things down
• Trim and/or shrink the databases if needed
• Install and test the version
• Apply load according to the plan
• Tune it and polish it up
• Interpret the results
• Go to production

The test environment is not equal to production: we apply coefficients

When it's hard to compare the hardware and the servers are connected over the same network channel, you can apply a coefficient determined from some standard request. In that case, you send N requests to the server and take the average time for the production server T and for the test server t. Then you calculate the coefficient with the formula k = T/t (production ÷ test). To adjust the result, you multiply the result obtained on the test server by this coefficient: R = k * t.

We apply coefficients in the same way when testing different versions

If the new version contains more functionality than the previous one (gets "heavier") or less (gets "lighter"), it can't be compared directly with its predecessor. In that case, transition coefficients are applied as well.

Load testing begins first and foremost with planning. A load testing plan made with JMeter consists of:

1. Defining the testing goals and developing load profiles.

2. Installing and configuring a bot-net for distributed testing (if needed).

3. Recording a Selenium CRUD test to use as the payload, to simplify preparing the test plan.

4. Setting up and debugging the load tests in JMeter.

5. Replaying the load tests many times according to the load profiles.

6. Analyzing the results and producing a report using the prepared template.

Configuring the jp@gc plugin to use PerfMon Metrics Collectors

To use the PerfMon Metrics Collectors modules from the jp@gc plugin, which collect load metrics on the server side, you also need to run the JMeter server agent on every server involved in the measurement (application servers, database servers, and so on). To do that, you need to:

1. Copy the serverAgent directory from the directory with the jp@gc plugins to the server whose load you need to measure.

2. On Windows servers, run the file startAgent.bat. On Linux servers, run startAgent.sh, after first granting execute permission with the command

3. chmod a+x startAgent.sh

Once the agent is running, the PerfMon Metrics Collector Listener modules become available for connecting to the agents. You can add several servers to them for monitoring. One graph can display different kinds of metrics: CPU, memory, disk subsystem, and network adapter load, and others.

To make it easier to create a JMeter test plan, use a set of Selenium end-to-end scenario tests, whose actions will be used to record the load tests. All the work with the tests can be done in the Thread Group branch of the test plan. A thread group is a group of threads. In load testing, each action gets its own autonomous thread, which the computer runs in parallel with the others.

To add a Thread Group, choose Add - Thread Group from the context menu of the Test Plan element. The Thread Group element lets you set the parameters of the load generated against the application. Its main parameters are:

1. Number of threads - the number of threads (simulated users working with the site at the same time);

2. Ramp-up period - the time interval at which the next thread is started;

3. Loop count - the number of times the scenario will run inside the Thread Group;

4. Forever - the scenario will run indefinitely until it is explicitly stopped;

5. Scheduler - a scheduler for the scenario's run time;

6. Action to be taken after a Sample Error - the action taken if a request causes an error.

Debugging the script means removing the various .jpg and .png files and links to third-party resources. In my script, more than half of the requests are such third-party resources: connections to various social networks, links to fonts on fonts.gstatic.com, and also the IT school site. All of this can be cleaned out (with the Delete key). In general, you can clean out the .js files too. The main thing is to find the request that carries your user's credentials in its body. And, for elegance, find the request that leads to the page where the user logs in. Together, they model the sequence of user actions "opened the page, then logged in".

Let's assume we were lucky enough to pick out the necessary requests. Or, well, we think we picked them out. Let's give them readable names! To check whether we picked out the right ones, we need to run the script.

If you need to authorize on the server for testing, add an HTTP Authorization Manager at the top of the tree (Thread Group – Add – Config Elements – HTTP Authorization Manager). It's intuitive. Enter the address of our resource there, with http:// (this is mandatory), along with the username and password.

If the Authorization Manager is needed to authorize on the server, then cookies are definitely used when logging in to the site. Add a Cookie Manager after the Authorization Manager (TestPlan – Add – Config Elements — HTTP Cookie Manager). You should probably always add this element whenever the user has to log in.

Recommended Minimum Set of Modules for a Test Plan

We recommend including the following modules in every new test plan (each of them has an online help system; just click the Help on this plugin button):

1. Add an HTTP Test Script Recorder to the WorkBench branch, for the initial recording of test scenarios, if the application works over the HTTP protocol.

2. Add a Uniform Random Timer randomizer with a value of 100-200 ms to the Test Plan branch.

3. Add HTTP Request Defaults to the Test Plan branch to configure default requests, if the application works over the HTTP protocol.

4. Add an HTTP Cookie Manager to the Test Plan branch if the application uses cookies.

5. Add the required number of Thread Groups to the Test Plan branch; they act as separate test cases into which the data from the proxy server should be recorded.

6. Add the standard tabular reports on the test results to the Thread Group branch: Summary Report, View Results in Table, View Results Tree.

7. Add the standard graphical reports on the test results to the Thread Group branch: Graph Results, Spline Visualizer.

8. Add the graphical reports of the jp@gc plugin to the Thread Group branch: jp@gc - Response Times Percentiles (shows data access percentiles), jp@gc - Response Times vs Threads(shows how the response time changes depending on the number of running threads), jp@gc - PerfMon Metrics Collector (for getting data on CPU, Memory, Swap, Disk I/O, Network I/O and other load from the server side, from the JMeter agents).

9. At the end of the Test Plan branch, add Monitor Results to track the state of the server during

the test run.

To keep this assignment simple, we won't use any scenario; we'll just put load on a set of the site's available pages. For that, we need to get a list (a map) of the site's URLs. Since modern sites have started using protection against scanning, it's convenient to use scanners through a browser add-on. For example, Link Klipper for Chrome. This app will scan the domain address and produce a list something like this:

http://software-testing.ru/user/forgot.php
http://software-testing.ru//user/user_add.php
https://top100.rambler.ru/top100/
http://software-testing.ru/app/
https://www.facebook.com/gorod.dp.ua/
...

Links to external sites, like the ones in red, should be deleted; otherwise they will spoil the test results with unreliable data. Then we do a global replace of the part 'http://test.local' with '" GET' (a quotation mark followed by a space), and we get

"GET /index.php

"GET /news/event-12.php

...

It's better to use Notepad++ for an operation like this, because Word adds special characters.

Note that the domain name is not included in the link, only the internal path. The domain itself is specified in the JMeter settings. Each link must be on a new line, and the document must be in plain text format.

1. Open ApacheJMeter.jar

2. On the left we see a tree of 2 nodes: TestPlan and Workbench (we forget about the second one right away; we won't need it). Right-click Test Plan and choose Add->Thread Group. (The interface has lots of features of varying usefulness, but we won't get distracted right now; we'll take the shortest path to our test, and later, if we feel like it, we'll study JMeter's extensive capabilities in more detail.)

1. Right-click Thread Group and add an Access Log Sampler (Thread Group->Add->Sampler->Access Log Sampler)
2. Fill in the data for access.log Enter the server address (domain name only) Choose the protocol: HTTP or HTTPS, entered in capital letters Set the port to 80 Specify the path to the access.log file on your PC

Now we add display tools to the test:

1. Thread Group->Add->Listener->View Results in Table

2. Thread Group->Add->Listener->Graph Results

3. Thread Group->Add->Listener->Aggregate Report

In View Results in Table, you need to fill in the Filename field (if you don't specify a path, the log file will be created next to jmeter.bat). You need to create the log for debugging, because JMeter's GUI doesn't show any useful information about errors.

The test plan is ready, so we move on to testing it :) and debugging (don't worry, it might well work on the first try).

File->Save, and do this every time after making changes to the test plan. This is important: JMeter sometimes freezes, and you end up having to restore the test from memory.

Run->Clear All (you can skip this the first time, but you'll need it later anyway).

Run->Start.

And we go look at View Results in Table. If we're lucky, there will be a single row with a green check mark in the Status column.

Before starting the test, let's add a random delay (Uniform Random Timer) of 0-1000 milliseconds at the beginning of the scenario; it usually helps smooth out the graphs a bit. As a result, the scenario works like this: it waits a random number of milliseconds, reads a line from the log, makes an HTTP request, passes the results to the listeners, waits again, reads the next line, and so on.

Let's do the first, trial test. In the thread group properties, set: Number of Threads (users): 100, Ramp-Up Period (in seconds): 100. We're going to unleash 100 virtual users on the site, sending them into battle one at a time over 100 seconds, that is, one user per second. I pulled the numbers 100 and 100 out of thin air, but you have to start somewhere.

Let us remind ourselves once more that we stand a good chance of slowing the site down or even bringing it down (which can be bad if it's a project that's already live). OK, being of sound mind and clear memory, and aware of our responsibility for our actions, we begin.

File->Save, Run->Clear All, Run->Start and we go look at Graph Results. We see, say, a picture like this

Let's run similar tests for another site. Now we'll figure out how to tell exactly when the moment of server overload arrives, using the View Table in Tree listener. We can determine that the first failures and 503 errors in the system occurred with 31-34 threads running. Consequently, the critical number of active users for the server is about 35. But keep in mind that the activity of test users is far higher than that of real people. It's unlikely anyone will click through a site at a rate of 1 page every 2 seconds.

1. The signal that the server is overloaded is that the average response time (Average) grows while the processing rate (Throughput) doesn't change. This means that somewhere on the server, operations are lining up in a queue and there isn't enough performance to serve all the requests. You can see this moment on the Graph Results graph. In that case, state in your conclusions the number of users N at which it happens.

2. The second metric is the appearance of server errors: 503, 505, 404, and so on. This means that, besides queuing requests and failing to process them in time, the server has started simply rejecting some of them. This is already a critical overload. You can observe it in the View Results table. Note this moment in the report.

3. The third metric is the error percentage at a given number of users. This parameter is in the Summary Report. We compare it with the requirements and put it in the report. The requirements must specify the acceptable error threshold.

4. A positive result. A tester's job isn't always just to find bugs; sometimes there are none. If the system withstood the planned load and the error percentage stayed within the norm set by the requirements, then the system successfully passes load testing, and we get PASS QA.

After analyzing these parameters, you can draw conclusions and write the load testing report. The key data in this report will be: the maximum number of users who can work without failures, the average request processing time in that mode, and the error percentage in the critical mode. Then this data needs to be compared with the requirements set for the project, and the appropriate conclusions drawn: will the system withstand the load or not.

2. Table of contents.

3. Introduction.

States what the document is and gives a brief description of the application under test, which can be taken, for example, from the technical specification.

States which main functions, subsystems, or modules of the application were tested.

Testing tools used: lists all the tooling used for testing, for example, apache-jmeter-2.7, JMeterPlugins-0.5.2, Selenium IDE-1.8.1.

Appendices: all testing artifacts, for example, the JMeter test plan, .jtl files with results, charts of the measurement results, screenshots, and so on.

Network path to the application: the network address or addresses at which the Application's interface can be accessed.

A schematic, graphical description of the Application's network structure, listing its main resources, nodes, and subsystems.

Inventory of the Application's resources. The application's resources shown in the diagram above are described in a table. The table has the columns: No., Resource, Hardware, Software.

7. Performance indicators. Lists the indicators measured at this stage and the metrics used for them. For example, the indicators given above.

8. Load profiles. Describes sets of operations with different load intensities and the conditions for running the test, for example, the required application settings, the number of concurrent users, the application subsystem under load, the test duration, and so on.

9. Performance requirements. Describes the acceptable values of the measured parameters at which the load test is considered successfully passed.

10. Description of the testing methodology. Briefly describes the approaches to testing and how the test environment was prepared, for example, the structure of the bot-net, the description of the payload, and the principle of selecting tests.