Server Going down frequently

[ramindia]

Hi all

I have dedicated server

every day the server goes down, we are not able to access ssh and WHM pannel
and server not serving any of the sites hosted in the server

i have made script to record how many hits coming to the server
so i got a report before server not accessble around 990

i have csf and lfd running, and i have set CT_LIMIT = 30

but still iam getting lot of hits from different servers, how can i block them

and give assurance to my clients all sites running smooth

any other suggestions to make server stable.

when i lost connection to all the access, i need to go to APC panel to restart server.



Ram

 

[cPanelTristan]

I would suggest adding sys-snap.sh and reviewing the processes, memory, and so on when the server goes down:

http://forums.cpanel.net/f5/server-...ull-since-last-update-246452.html#post1019842

Additionally, you may want to check /var/log/messages during the time the machine is not functional to see what it shows for process(es). If any kernel issues are happening, you'll see those logged to /var/log/messages location.

After that, try checking sar output to see if you might be running out of memory or have high I/O

You should be able to get an idea of the time(s) of the occurrence using "last | grep boot" in command line, since that will show each reboot on the machine.

 

[ramindia]

Hi

thanks

since 24 hours the server got hanged

i am running sys snap, waiting for the server halt to take output of sys snaps

when the machine is not functioning, iam not able to access, but later when i look at /var/log/message
i see maximum hits to http and load high on 287 * on 4 cpus and memory also using 80% of 4GBRAM

iam looking sar report if i can see any clue why the server going down frequently

last | grep boot
reboot system boot 2.6.18-194.26.1. Tue Dec 20 06:27 (17:09)
reboot system boot 2.6.18-194.26.1. Thu Dec 15 06:06 (5+17:30)
reboot system boot 2.6.18-194.26.1. Sun Dec 11 23:38 (8+23:57)
reboot system boot 2.6.18-194.26.1. Sun Dec 11 05:52 (9+17:44)
reboot system boot 2.6.18-194.26.1. Mon Dec 5 00:27 (15+23:09)
reboot system boot 2.6.18-194.26.1. Sat Dec 3 00:01 (17+23:35)
reboot system boot 2.6.18-194.26.1. Fri Dec 2 01:16 (18+22:20)
reboot system boot 2.6.18-194.26.1. Fri Nov 11 14:37 (39+08:58)
reboot system boot 2.6.18-194.26.1. Fri Nov 11 14:57 (00:-20)
reboot system boot 2.6.18-194.26.1. Mon Nov 7 00:19 (4+14:17)
reboot system boot 2.6.18-194.26.1. Wed Nov 2 01:04 (9+14:32)
reboot system boot 2.6.18-194.26.1. Tue Nov 1 01:43 (10+13:53)
reboot system boot 2.6.18-194.26.1. Thu Oct 27 01:19 (5+00:21)
reboot system boot 2.6.18-194.26.1. Tue Oct 25 07:49 (6+17:51)
reboot system boot 2.6.18-194.26.1. Mon Oct 24 07:07 (7+18:34)
reboot system boot 2.6.18-194.26.1. Thu Oct 20 10:17 (11+15:24)
reboot system boot 2.6.18-194.26.1. Thu Oct 20 10:13 (00:00)
reboot system boot 2.6.18-194.26.1. Fri Sep 23 02:57 (27+07:16)
reboot system boot 2.6.18-194.26.1. Fri Sep 23 02:54 (00:00)
reboot system boot 2.6.18-194.26.1. Thu Sep 22 08:42 (17:58)
reboot system boot 2.6.18-194.26.1. Wed Feb 23 14:17 (211+11:24)
reboot system boot 2.6.18-194.26.1. Tue Dec 28 01:02 (57+12:47)
reboot system boot 2.6.18-194.26.1. Mon Dec 20 14:38 (64+23:11)
reboot system boot 2.6.18-194.26.1. Mon Dec 20 22:55 (-8:-18)
reboot system boot 2.6.18-194.26.1. Thu Dec 16 00:42 (4+13:54)
reboot system boot 2.6.18-194.26.1. Thu Dec 16 00:31 (4+14:05)
reboot system boot 2.6.18-194.26.1. Thu Dec 16 00:24 (00:01)
reboot system boot 2.6.18-194.26.1. Thu Dec 16 00:20 (00:02)
reboot system boot 2.6.18-194.26.1. Thu Dec 16 00:09 (00:08)



Support
Main >> Security Center >> Scan for Trojan Horses
Scan for Trojan Horses
Appears Clean

/dev/core
/dev/stderr

Scanning for Trojan Horses.....
.

Possible Trojan - /usr/sbin/pureauth
.
.
.
.

Possible Trojan - /etc/cron.daily/logrotate
.

Possible Trojan - /usr/bin/cpan
.
.

Possible Trojan - /usr/bin/instmodsh
.

Possible Trojan - /usr/bin/prove
.

Possible Trojan - /usr/bin/xsubpp
.
.
.
6 POSSIBLE Trojans Detected


any suggestions

Ram

 

[faisikhan]

Hi,

1. First of all "Server Reboot" isn't the permanent solution of any tiny or big issue, you may get rid of the issue at run time just rebooting the server but you won't get rid of it permanently.
2. So you have to make your server secure to avoid unwanted hits because there may be some security hole so follow that link 20 Linux Server Hardening Security Tips
3. As cPanelTristan said tail -f /var/log/messages will help you a lot to get the root cause of the unwanted hits coming from different IP addresses & so block those IPs.
4. Run top command to see the process which is consuming much memory and cpu resources and restart it once before you reboot the server like httpd in your case.
5. Check the available memory upon the server and extend it if possible or remove swapping processes Find out what is using your swap | All things Sysadmin

 

[ramindia]

Hi faiskhan

thanks for quick reply and advice

1. iam rebooting the server only when iam not able to access the server for several hours.

2. most of the packages are removed, if they are not using.

3. iam getting some different IP at a time like 900hits, that time iam not able to access the server at all see who is hitting me, so i have added in LFD CT_LIMIT=30 to block. still the server unreachable some time ( so i force to reboot, if i get access sure iam doing restart http, so i see the load go down fast.)

4. top -c always show more process running by httpd, thats due to attacks

5. Swap is always free only, but i see sometime memory occupied.

i will try some of your tips.

i welcome any more suggestion to address my problem to fix permanently.

thanks

Ram

 

[cPanelTristan]

So do you believe you are getting a denial of service attack on Apache? If so, then you might want to review this thread and also check MaxClients as noted in the thread:

http://forums.cpanel.net/f5/how-restart-apache-trough-cpanel-command-line-187252.html#post766101

 

[ramindia]

Hi cPanelTristan

yes i see SYN Attacks

[error] server reached MaxClients setting, consider raising the MaxClients setting

the site normally load was not more than 100connections
but i have set to 512 connections

when the last flood happens it went to 1024 connections

I have enabled CT_LIMIT=30

I have enabled system-snapshot

before server stopped, i took the log, below one ( IP and other information replaced due to security problems)


File attached (cpanel-forum.txt)


After i have hard rebooted the server

I have enabled the SYN Flood config in CFD/LFD

SYNFLOOD = "1"
SYNFLOOD_RATE = "100/s"
SYNFLOOD_BURST = "150"

CONNLIMIT = "80;10"


Iam monitoring the server

any other suggestion, what iam missing

Ram

 

[faisikhan]

Hi

It is good that you are monitoring the server very closely & from the file attached I have seen that the problem is caused by the unwanted IP hits causing the server down and consuming other resources. So I hope so to block those IP addresses & filtering the network traffic(via http or https) will prevent the server going down. Did you define any IP-tables rule for that before, if not see here for filtering the network traffic /http://www.thegeekstuff.com/2011/06/iptables-rules-examples/ , also see No.23 for DoS attack.

 

[ramindia]

Hi faiskhan

i have added

I have enabled the SYN Flood config in CFD/LFD

SYNFLOOD = "1"
SYNFLOOD_RATE = "100/s"
SYNFLOOD_BURST = "150"

CONNLIMIT = "80;10"

Looks server stable past 4days

and server still under monitoring 24hours basis
to see if any problem repeats again

Ram

 

[ruzbehraja]

• How many sites are hosted on that server?
• Is the attack on a particular site(s)?
• Do you have mod_security installed and enabled?

 

[ramindia]

Hi

2 sites are hosted on that server

only 1 domain getting attacked

iam not sure "Do you have mod_security installed and enabled?"

Ram

 

[ruzbehraja]

You may want to do this: http://forums.cpanel.net/f5/my-serv...w-can-i-prevent-attack-147781.html#post626445

Also see: Security FAQ,

 

[ramindia]

Hi

thanks for the suggestion

as per the document here is my config


SYN:


more /proc/sys/net/ipv4/tcp_synack_retries
5

more /proc/sys/net/ipv4/tcp_max_syn_backlog
1024
more /proc/sys/net/ipv4/tcp_syncookies
1

Spoofing :

more /proc/sys/net/ipv4/conf/*/rp_*
::::::::::::::
/proc/sys/net/ipv4/conf/all/rp_filter
::::::::::::::
0
::::::::::::::
/proc/sys/net/ipv4/conf/default/rp_filter
::::::::::::::
1
::::::::::::::
/proc/sys/net/ipv4/conf/eth0/rp_filter
::::::::::::::
1
::::::::::::::
/proc/sys/net/ipv4/conf/lo/rp_filter
::::::::::::::
0



iptables rules


16 CONNLIMIT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 flags:0x17/0x02 #conn/32 > 10



Chain SYNFLOOD (1 references)
num target prot opt source destination
1 RETURN all -- 0.0.0.0/0 0.0.0.0/0 limit: avg 100/sec burst 150
2 LOG all -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix `Firewall: *SYNFLOOD Blocked* '
3 DROP all -- 0.0.0.0/0 0.0.0.0/0