Stress Testing

What factors affect the stress test results?

• Network latency from the stress machine to the server (It is recommended to test within the local network or on the same machine)
• Bandwidth from the stress machine to the server (It is recommended to test within the local network or on the same machine)
• Whether HTTP keep-alive is enabled (It is recommended to enable it)
• Whether the concurrency level is sufficient (For external network testing, it's advisable to enable higher concurrency)
• Whether the number of server processes is reasonable (For a helloworld service, it is recommended that the number of processes matches the number of CPUs; for database services, the number of processes should be four times the number of CPUs or more)
• The performance of the business itself (e.g., whether an external database is used)

What is HTTP keep-alive?

The HTTP Keep-Alive mechanism is a technique for sending multiple HTTP requests and responses over a single TCP connection, which significantly impacts performance test results. Disabling keep-alive may cause QPS to drop dramatically.
Currently, browsers default to keeping connections alive, meaning that after a browser accesses a specific HTTP address, it temporarily retains the connection for reuse during subsequent requests, improving performance.
It is recommended to enable keep-alive during stress testing.
Additionally, if keep-alive is not enabled during testing, the client's local ports can quickly run out due to timewait states, resulting in failed requests once the total request count exceeds a certain limit (generally around 28,000).

How to enable HTTP keep-alive during stress testing?

If you are using the ab program for testing, you need to add the -k parameter, for example: ab -n100000 -c200 -k http://127.0.0.1:8787/.
apipost needs to return a gzip header in the response to enable keep-alive (a bug in apipost, see below).
Other stress testing programs usually enable keep-alive by default.

Why is the QPS so low when testing from the external network?

It is normal to see low QPS due to significant external latency. For example, testing a baidu page may yield a QPS of only a few dozen.
It is recommended to conduct tests within the local network or on the same machine to eliminate network latency effects.
If you must test on the external network, you can increase the concurrency level to enhance throughput (provided that sufficient bandwidth is available).

Why does performance decline after proxying through nginx?

Nginx consumes system resources. Additionally, the communication between nginx and webman also requires some resources.
However, system resources are limited, and webman cannot access all system resources. Therefore, it is normal for the overall system performance to decline.
To minimize the performance impact of nginx proxying, consider disabling nginx logging (access_log off;), and enabling keep-alive between nginx and webman. See nginx proxy for reference.

Additionally, https consumes more resources compared to http, as https requires SSL/TLS handshake, data encryption and decryption, and larger packet sizes that consume more bandwidth, all of which can lead to performance degradation.
If short connections (without HTTP keep-alive) are used during testing, each request will incur additional SSL/TLS handshake communications, significantly reducing performance. It is recommended to enable HTTP keep-alive when testing https.

How to know if the system has reached its performance limit?

Generally, when CPU reaches 100%, it indicates that the system performance has reached its limit. If there are still idle CPU resources, it means that the limit has not been reached, and at this point, you can appropriately increase concurrency to improve QPS.
If increasing concurrency does not boost the QPS, it could indicate insufficient webman processes; consider increasing the number of webman processes. If performance still does not improve, check if bandwidth is sufficient.

Why is my test result showing that webman performs worse than the go gin framework?

techempower benchmarks show that webman outperforms gin by nearly twice in all metrics, including plaintext, database queries, and database updates.
If your results differ, it may be because you are using ORM in webman, which incurs a significant performance penalty. Consider comparing webman + native PDO with gin + native SQL.

How much performance loss occurs when using ORM in webman?

Here is a set of benchmark data.

Environment
Alibaba Cloud server with 4 cores and 4GB RAM, local MySQL database, randomly querying one record from 100,000 rows and returning JSON, tested from the local machine.

Using native PDO
webman QPS is 17,800

Using laravel's Db::table()
webman QPS drops to 9,400

Using laravel's Model
webman QPS drops to 7,200

Results with thinkORM are similar, with little difference.

Tip
Although using ORM incurs a performance hit, for 99% of businesses, the performance is still significantly sufficient. If you happen to be in that 1%, you can easily resolve it by adding more CPUs or servers.
We should find a balance among development efficiency, maintainability, and performance metrics, rather than blindly pursuing performance.

Why is the QPS so low when testing with apipost?

The apipost stress testing module has a bug; if the server does not return a gzip header, keep-alive cannot be maintained, resulting in significant performance degradation.
The solution is to compress the data and add a gzip header in the response, for example:

<?php
namespace app\controller;
class IndexController
{
    public function index()
    {
        return response(gzencode('hello webman'))->withHeader('Content-Encoding', 'gzip');
    }
}

Furthermore, apipost may underperform in some cases; under the same concurrency, using apipost may yield around 50% lower QPS than ab.
It is recommended to use ab, wrk, or other professional stress testing software instead of apipost.

Setting an Appropriate Number of Processes

Webman defaults to opening four times the number of CPU processes. In fact, for a helloworld service without network I/O, using a number of processes equal to the number of CPU cores is most optimal, as it minimizes process switching overhead.
If dealing with blocking I/O services involving databases or Redis, the number of processes can be set to three to eight times the number of CPUs, as more processes are needed to improve concurrency, while process switching overhead is relatively negligible compared to blocking I/O.

Reference ranges for stress testing

Cloud server 4 cores 4GB 16 processes, local machine/internal network testing

Third-party techempower benchmark data

Example Stress Test Commands

ab

# 100000 requests with 200 concurrency, keep-alive enabled
ab -n100000 -c200 -k http://127.0.0.1:8787/

# 100000 requests with 200 concurrency, keep-alive disabled
ab -n100000 -c200 http://127.0.0.1:8787/

wrk

# 200 concurrency testing for 10 seconds, keep-alive enabled (default)
wrk -c 200 -d 10s http://example.com