Part 2 — Performance Goals and k6 Metrics
Mostafa Moradian · Follow
Published in · 5 min read · Jun 19, 2019
--
Performance testing is an umbrella term for a group of tests that encompasses many types of tests, as discussed in the first part of this series. Each test type tries to answer a set of questions to make the performance testing process more goal-oriented. This means that just running tests is not enough, you have to have a set of goals to reach.
Since we’re testing an API or a website, the following goals may be relevant, in which you can choose either one or more:
- Concurrency: systems which would set this goal, usually have a concept of end-user and need to see how the system behaves while many concurrent users try to access the system. They basically want to test how many of requests fail/pass under high loads of users. This both includes many concurrent users and each requesting multiple resources at the same time.
- Throughput: systems with no concept of end-users, would set this goal to see how the system behaves overall, while there is a ton of requests/responses coming in/out of the system.
- Server response time: this goal signifies the time it takes from the initial request from the client to the server up until a response is sent back from the server.
- Regression testing: sometimes the goal is not to put “heavy load” on the system, but is more about “normal load” and functional and regression testing to see how a change would affect our system’s performance and if it still adheres to our defined SLAs.
The general idea is to measure how a system or system of systems behave(s) under heavy load, in terms of speed, scalability, stability and resiliency. Each of which can be measured by these goals.
- Speed can be measured by time it takes for request to be handled by the server and how much time it takes for this request/response to happen.
- Scalability can be measured by how well the system scales if the load is increased and by measuring if it sustains over a period of time under this load.
- Stability can be measured by how well the system sustains the load and to see if it stands against a high number of errors and events and still stays responsive and stable.
- Resiliency can be measured by how the system recovers from crashes and down-times and responds to requests, after putting too much or too frequent load on it and eventually crashing the system.
Rerunning Tests to Verify the Results
You can rerun the tests to see if they hold almost the same results during different tests and compare the tests to see if they deviate.
If they are almost the same, you can analyze the tests and derive your results, otherwise you should pinpoint where it deviates and try to find a way to prevent it from happening, like a bottleneck.
k6 and the Metrics
k6 supports a set of built-in and custom metrics that can be used to measure various things and to either achieve the above mentioned goals or prove them wrong. The metrics that can be used to define custom metrics are: Counter, Gauge, Rate and Trend.
As you’ve probably seen above, these following tables describes reported built-in metrics, present on all tests:
Custom (non-built-in) Metrics
1. Counter
This is a simple cumulative counter that can be used to measure any cumulative value like number of errors during the test.
As you can see in the above example, it counts the number of 404 errors that are returned by the test. The result is evident in the screenshot below:
Since it is a beginner’s guide, I try to stick with simple examples, but you can extend and customize them to your specific case.
2. Gauge
This metric lets you keep the last thing that is added to it. It’s a simple over-writable metric that holds its last added value.
This metric can be used to retain the last value of any test item, be it response time, delay or any other user-defined value.
If you run the following code, you’ll see that it catches the latest error code, which is 404.
The result of the test is presented in the screenshot below.
3. Rate
This built-in metric keeps the rate between non-zero and zero/false values. For example if you add two false and one true value, the percentage becomes 33%.
It can be used to keep track of the rate of successful request/responses and compare them with errors.
In the following piece of code, you can see that I added res.error_code as a measure to see how many errors I’ll catch.
Below is the result of the test, which is 100% errors.
4. Trend
This metric allows you to statistically calculate your custom value. It will give you minimum, maximum, average and percentiles, as is evident in the above screenshots for http_req* requests.
The above example of trend metric shows how to calculate the sending and receiving time without taking into account the waiting time. The result is shown below:
In this part, I’ve tried to describe the goals of performance testing and how one can use metrics to achieve those goals. In the next sections, I’ll try to go more in-depth and present you more details on how to define custom metrics and how to use them.
Now that you have a good grasp of performance goals and k6 metrics, you can move to the next article, in which I try to show you how to write and run a k6 script.
