Server Slowdown: Only Resolved by Rebooting

[psytanium]

I'm encountered an issue with my server's performance. Out of the blue, it slowed down significantly. After contacting my hosting provider, they assured me that everything looks fine on their end. I've inspected hardware monitoring, which shows that I'm using only 15% CPU, 60% HDD capacity, 40% RAM, and so on.

What confuses me is that the problem vanishes for a brief period (about 1 hour) each time I reboot my server.

I've also checked the processes and system health, but nothing appears to be exceeding its limits.

Could anyone help me identify the root cause of this issue? Your insights would be greatly appreciated.

Many thanks

 

[cPRex]

Hey hey! These are always tricky to diagnose when it isn't happening.

When you see the slowness is it *everything* or just websites? As in, does cPanel & WHM load slowly, response on the command line, etc?

I'd check the Apache status with "apachectl status" to see what that shows during the time of slowness.

 

[psytanium]

All the websites load very slow, but the cPanel and WHM load normally.

 

[cPRex]

Definitely check the Apache scoreboard with that command, then, to confirm you aren't experiencing a DoS of some sort.

 

[psytanium]

I will run the command "apachectl status" during the time of slowness, but now the server is working fine, this is the output :

apachectl status
Apache Server Status for localhost (via ::1)

Server Version: Apache/2.4.57 (cPanel) OpenSSL/1.1.1v mod_bwlimited/1.4

Server MPM: prefork

Server Built: Aug 15 2023 22:58:51

   --------------------------------------------------------------------------

   Current Time: Monday, 11-Sep-2023 18:24:28 EEST

   Restart Time: Sunday, 10-Sep-2023 17:58:20 EEST

   Parent Server Config. Generation: 33

   Parent Server MPM Generation: 32

   Server uptime: 1 day 26 minutes 7 seconds

   Server load: 1.86 1.38 1.13

   Total accesses: 457702 - Total Traffic: 12.5 GB - Total Duration:
   319348508

   CPU Usage: u9.92 s65.66 cu62875.5 cs16596.5 - 90.4% CPU load

   5.2 requests/sec - 149.2 kB/second - 28.7 kB/request - 697.721 ms/request

   113 requests currently being processed, 14 idle workers

 RRR_RRRKRRRK_R_RWRRRKKRRRRKRRRRRRRRRRRRRRRRRRR_RKRRRRR__RRRRRRRR
 RRR.RRRRRRRRRRWRRRR.R_KRR_RK.R_RRRR.RRRRRKW_RRRRR_R_R_RRCRR.....
 .RR..R...RR_..RR......

   Scoreboard Key:
   "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
   "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
   "C" Closing connection, "L" Logging, "G" Gracefully finishing,
   "I" Idle cleanup of worker, "." Open slot with no current process

I have around 75 websites hosted on this VPS server. From the current output, do you see anything wrong ?

 

[cPRex]

That's a lot of reading requests. I'd wait until the server slows down and run the same command, as now you have something to compare it to. If you see the scoreboard, the section with the Rs and Ks, is full during the time of slowness, it is likely some type of attack. You can get more details here:

How to Mitigate Slowloris Attacks | cPanel & WHM Documentation

This document provides several methods to mitigate the impact of Slowloris attacks.

docs.cpanel.net

or you can check "apachectl fullstatus" to see exactly what pages are being accessed at that time.

 

[psytanium]

That's a lot of reading requests. I'd wait until the server slows down and run the same command, as now you have something to compare it to. If you see the scoreboard, the section with the Rs and Ks, is full during the time of slowness, it is likely some type of attack. You can get more details here:

How to Mitigate Slowloris Attacks | cPanel & WHM Documentation

This document provides several methods to mitigate the impact of Slowloris attacks.

docs.cpanel.net

or you can check "apachectl fullstatus" to see exactly what pages are being accessed at that time.

1. I installed mod_reqtimeout module
2. The file " /etc/apache2/conf.modules.d/375_mod_reqtimeout.conf" is created with the right content
3. But I didn't understand this step: Place any configurations that you wish to use the mod_reqtimeout module in the /etc/apache2/conf.d directory.
   Your include file should resemble the following configuration to mitigate Slowloris attacks:

<IfModule mod_reqtimeout.c>
RequestReadTimeout header=20-40,MinRate=500 body=20-40,MinRate=500
</IfModule>

 

[cPRex]

I wouldn't make any changes until we see the output of that command from the server in a slowed state. At this point, we're just guessing.

 

[SimpleSonic]

If it is a DoS attack like Slowloris, you may want to consider using LiteSpeed web server instead of Apache as it will easily handle these types of attacks without burning up precious CPU and RAM as well as making your web server faster in general.

However, whether or not to use LiteSpeed would obviously depend on your budget.

 

[psytanium]

cPRex, you linked this document: How to Mitigate Slowloris Attacks | cPanel & WHM Documentation

The recommend method is using mod_reqtimeout by applying those 3 steps:

1. I installed mod_reqtimeout module
2. The file " /etc/apache2/conf.modules.d/375_mod_reqtimeout.conf" is created with the right content

I didn't understand the 3rd step, which indicate to place any configurations that you wish to use the mod_reqtimeout module in the /etc/apache2/conf.d directory.

Your include file should resemble the following configuration to mitigate Slowloris attacks:

<IfModule mod_reqtimeout.c>
RequestReadTimeout header=20-40,MinRate=500 body=20-40,MinRate=500
</IfModule>

How can I include the configurations ?

Thanks for your help

 

[cPRex]

Honestly, I don't like how that's worded either so I'm not 100% sure. I'd need to reach out to a developer for clarification.

But at this point, don't make any changes - we want to confirm *why* the system is in the slow state first before we start guessing about what may need to be fixed.