Stress and performance test #8
Description
In GitLab by @mhxion on Dec 1, 2023, 24:02
I did some digging into why the responses are slower all on a sudden.
- The timeout issue can be solved with just increasing the timeout to 180s (previously it was 60s). In itself, this isn't a big issue. But the following might be related to it.
- The responses are slower than before. Querying 2500 users from dev-002 takes 6-7 minutes where just a week ago it was taking 4-5 minutes. Querying 600+ users from production also takes 5-6 minutes where it should take at most 2 minutes. These timings remain the same even when I run bill-teams on the servers through SSH!
Monitoring server load
Since no. 2 wasn't a latency issue, I wanted look into the server load. This can also count as a minimal stress test.
When I run bill-teams targeting dev-002 on my own machine and monitor htop stat on dev-002 through SSH, I get the following results. Worth noting, the slow responses mainly come from when we make asynchronous requests to get every user data.
i. Making 100 concurrent connections (so that's 100 concurrent GET requests to the user endpoint) gives us the following the htop change.
The number 188 for threads make sense (before starting bill-teams it was about 86 threads). That many concurrent connections give us 100% CPU usage. This is the default number of concurrent connections of elAPI which gave us really satisfying performance before on dev-002.
ii. Making 5 concurrent connections gives us the following htop.
That also causes almost 100% CPU usage spike. In fact, having 100 concurrent connections doesn't improve the overall timing by that much. I should have done this test long ago so I would have some more data to compare :/ I am not sure how bad this is.