In Progress CPANEL-40292 - testing installation and uninstallation of Nginx sometimes leave Nginx in a error wont start

[Spirogg]

Ive hit this issue 2 times already

Has anyone else had this issue while testing Ubuntu cPanel 102.08


Ive installed Nginx then clciked restart Nginx and it works. then Uninstall Nginx soft reboot server and install again. then click restart Nginx and get this error

  Error: [2022-03-16 22:13:18 -0500] info [restartsrv_nginx] systemd failed to start the service “nginx” (The “/usr/bin/systemctl restart nginx.service --no-ask-password” command (process 1898) reported error number 1 when it ended.): Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details. Waiting for “nginx” to start ………failed. Cpanel::Exception::Services::StartError Service Status undefined status from Cpanel::ServiceManager::Services::Nginx Service Error (XID nnxpmc) The “nginx” service failed to start. Startup Log Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use) Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use) Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use) Mar 16 22:13:18 server2.e.com nginx[1899]: nginx: [emerg] still could not bind() Mar 16 22:13:18 server2.e.com systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Mar 16 22:13:18 server2.e.com systemd[1]: nginx.service: Failed with result 'exit-code'. Mar 16 22:13:18 server2.e.com systemd[1]: Failed to start nginx - high

I also get an email that it failed

The service “nginx” appears to be down.

Server server2 Primary IP Address xxx.xx.xx.xx Service Name nginx Service Status failed Notification The service “nginx” appears to be down. Service Check Method The system’s command to check or to restart this service failed. Number of Restart Attempts 6 Service Check Raw Output (XID zup3y4) The “nginx” service is down. The subprocess “/usr/local/cpanel/scripts/restartsrv_nginx” reported error number 255 when it ended. Startup Log Mar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) Mar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use) Mar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) Mar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use) Mar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) Mar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use) Mar 16 20:04:32 server2 nginx[5588]: nginx: [emerg] still could not bind() Mar 16 20:04:32 server2 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Mar 16 20:04:32 server2 systemd[1]: nginx.service: Failed with result 'exit-code'. Mar 16 20:04:32 server2 systemd[1]: Failed to start nginx - high performance web server. Log Messages The system could not provide log messages for “nginx” because it failed to read all of the potential log files with the following errors: Error while attempting to open “/var/log/messages”: “No such file or directory”, Error while attempting to open “/var/log/secure”: “No such file or directory” Memory Information Used 870 MB Available 10.79 GB Installed 11.64 GB Load Information 4.40 1.47 0.60 Uptime 41 minutes and 13 seconds IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 0.71 0.03 0.43 2.12 0.00 96.71 Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd dm-0 21.63 555.17 119.17 0.00 1372508 294624 0 loop0 0.02 0.14 0.00 0.00 338 0 0 loop1 0.02 0.14 0.00 0.00 338 0 0 loop2 0.02 0.14 0.00 0.00 350 0 0 loop3 0.60 0.72 0.00 0.00 1788 0 0 loop4 0.02 0.43 0.00 0.00 1074 0 0 loop5 0.02 0.14 0.00 0.00 336 0 0 loop6 7.25 7.37 0.00 0.00 18219 0 0 loop7 0.03 0.44 0.00 0.00 1083 0 0 loop8 0.47 0.80 1.40 0.00 1976 3456 0 loop9 0.03 0.14 0.00 0.00 349 8 0 sda 15.39 558.98 118.39 0.00 1381917 292692 0 scd0 0.08 2.97 0.00 0.00 7336 0 0 Top Processes PID​ Owner​ CPU %​ Memory %​ Command​ 5929 root 0.46 0.11 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/bin/leechprotect 6297 root 0.38 0.03 /lib/systemd/systemd-udevd 1141 mysql 0.36 1.29 /usr/sbin/mysqld 1087 cpanelconnecttrack 0.36 0.03 /usr/local/cpanel/3rdparty/sbin/p0f -i any -u cpanelconnecttrack -d -s /var/cpanel/userhomes/cpanelconnecttrack/p0f.socket less 400 and not dst port 80 and not dst port 443 and tcp[13] & 8==0 1137 root 0.34 1.07 /usr/local/cpanel/3rdparty/perl/532/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 --pidfile=/var/run/spamd.pid --listen=5 --listen=6 The chkservd process attempts to connect to “127.0.0.1:80” in order to validate that this service is functioning. If you blocked connections with iptables or the “Host Access Control” interface in WHM, this failure may be a false positive. To resolve this issue, either open the firewall to allow connections as the root user to “127.0.0.1:80” or disable checks for this service in WHM’s “Service Manager” interface with the “Configure Monitor Settings” link below. The system generated this notice on Thursday, March 17, 2022 at 1:13:44 AM UTC. “Service failures (ChkServd)” notifications are currently configured to have an importance of “High”. You can change the importance or disable this type of notification in WHM’s Contact Manager at: Do not reply to this automated message.

then I uninstalled and reinstalled and it starts ?

also the first time it removed POP3 and IMAP in service Manager
(Both selections = enable and monitor check marks) ?
and got LMTP Error Failed email. (second time Nginx would not start everything else seems to be intact. IMAP POP3 LMTP

The service “lmtp” appears to be down.

Server server2 Primary IP Address xxx.xx.xxx.xxx Service Name lmtp Service Status failed Notification The service “lmtp” appears to be down. Service Check Method The system’s command to check or to restart this service failed. Number of Restart Attempts 1 Service Check Raw Output (XID xnkcbd) The “dovecot” service is down. The subprocess “/usr/local/cpanel/scripts/restartsrv_lmtp” reported error number 255 when it ended. Startup Log No startup log Log Messages The system could not provide log messages for “lmtp” because it failed to read all of the potential log files with the following errors: Error while attempting to open “/var/log/messages”: “No such file or directory”, Error while attempting to open “/var/log/secure”: “No such file or directory” Memory Information Used 792 MB Available 10.86 GB Installed 11.64 GB Load Information 0.25 0.59 0.47 Uptime 8 minutes and 34 seconds IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 1.56 0.03 1.04 7.65 0.00 89.71 Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd dm-0 75.27 2169.09 211.83 0.00 1113416 108732 0 loop0 0.08 0.66 0.00 0.00 338 0 0 loop1 0.11 0.68 0.00 0.00 350 0 0 loop2 0.08 0.66 0.00 0.00 338 0 0 loop3 2.91 3.48 0.00 0.00 1788 0 0 loop4 0.10 2.09 0.00 0.00 1074 0 0 loop5 0.08 0.65 0.00 0.00 336 0 0 loop6 1.39 3.84 3.27 0.00 1969 1676 0 loop7 34.92 35.49 0.00 0.00 18219 0 0 loop8 0.13 2.11 0.00 0.00 1083 0 0 loop9 0.14 0.68 0.02 0.00 349 8 0 sda 53.99 2186.99 210.53 0.00 1122605 108068 0 scd0 0.16 6.12 0.00 0.00 3144 0 0 Top Processes PID​ Owner​ CPU %​ Memory %​ Command​ 1121 root 1.75 1.07 /usr/local/cpanel/3rdparty/perl/532/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 --pidfile=/var/run/spamd.pid --listen=5 --listen=6 2057 root 0.76 1.02 spamd child 795 root 0.67 0.31 /usr/lib/snapd/snapd 1068 mysql 0.60 1.31 /usr/sbin/mysqld 2093 root 0.44 0.03 /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf The chkservd process attempts to connect to “127.0.0.1:/var/run/dovecot/lmtp” in order to validate that this service is functioning. If you blocked connections with iptables or the “Host Access Control” interface in WHM, this failure may be a false positive. To resolve this issue, either open the firewall to allow connections as the root user to “127.0.0.1:/var/run/dovecot/lmtp” or disable checks for this service in WHM’s “Service Manager” interface with the “Configure Monitor Settings” link below. The system generated this notice on Thursday, March 17, 2022 at 1:59:14 AM UTC. “Service failures (ChkServd)” notifications are currently configured to have an importance of “High”. You can change the importance or disable this type of notification in WHM’s Contact Manager at: Do not reply to this automated message.

@cPRex

Have you ever run into this issue where nginx won’t start even if you click restart
You have to totally uninstall and reinstall

also if there ever was this issue on a production server. How would this affect users if you had to uninstall and reinstall
Would it ruin there websites. Or you just clear cache and it’s all good ?
Thanks

Kind Regards,
Spiro

 

[cPRex]

Hey there! I did the following:

-built an Edge server on Ubuntu
-installed Nginx through the EA4 interface

At this point it restarted normally

-removed Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager

-reinstalled Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager - no odd restart issues either

At this point, I can't reproduce, but you're welcome to create a ticket with our team if you'd like us to check out that specific machine.

 

[Spirogg]

Hey there! I did the following:

-built an Edge server on Ubuntu
-installed Nginx through the EA4 interface

At this point it restarted normally

-removed Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager

-reinstalled Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager - no odd restart issues either

At this point, I can't reproduce, but you're welcome to create a ticket with our team if you'd like us to check out that specific machine.

- uninstall and
- soft reboot server
and then install Nginx again is what I did,

I don't think it happens all the time.
possibly Not waiting till the process finished before soft reboot ?
not sure.
will try it again later and see if I can nail down the replication and if it happens I will open ticket with the team

Thanks

 

[Spirogg]

Hey there! I did the following:

-built an Edge server on Ubuntu
-installed Nginx through the EA4 interface

At this point it restarted normally

-removed Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager

-reinstalled Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager - no odd restart issues either

At this point, I can't reproduce, but you're welcome to create a ticket with our team if you'd like us to check out that specific machine.

ok It happened again I have Proxmox Server and this is a VM Ubuntu and cPanel 102.08 everything was running fine with serer for a few days and today i updated the Proxmox server some apt updates and installed them and had to reboot server. then I got email that Nginx is down. went to Home/Software/Nginx Manager click reboot Nignx and got the error again, So I just put in a ticket for them to look at why this is happening only in Ubuntu. the Other 2 VM's are using AlmaLinux and cPanel 102.08 and they are up and running. ?

here is the Ticket #94427991
if you want to Follow along
Thanks
Spiro

 

[cPRex]

Thanks so much - following now!

 

[Spirogg]

Thanks so much - following now!

@cPRex

Once again David H. has replicated this issue on a test server and found the issue


It appears that on Ubuntu servers, the "nginx-debug" service is set to be enabled when the ea-nginx package is installed. You can see this here:

[18:32:12 server2 root@94427991 ~]cPs# systemctl status nginx-debug
● nginx-debug.service - nginx - high performance web server
     Loaded: loaded (/lib/systemd/system/nginx-debug.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-03-22 01:37:20 CDT; 17h ago
       Docs: nginx documentation
    Process: 1001 ExecStart=/usr/sbin/nginx-debug -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
   Main PID: 1289 (nginx-debug)

On CentOS servers, this service is disabled as expected:

# systemctl status nginx-debug
● nginx-debug.service - nginx - high performance web server
   Loaded: loaded (/usr/lib/systemd/system/nginx-debug.service; disabled; vendor preset: disabled)

This "nginx-debug" process starts before "nginx", and binds to ports 80 and 443 as per usual. When "nginx" attempts to start, it fails as the ports are already bound to "nginx-debug". You can see this here in the server's log as well:

Mar 22 01:37:12 server2 systemd[1]: Starting nginx - high performance web server...
Mar 22 01:37:12 server2 systemd[1]: Starting nginx - high performance web server...
Mar 22 01:37:15 server2 nginx[1002]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Mar 22 01:37:15 server2 nginx[1002]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Mar 22 01:37:15 server2 nginx[1002]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)

I have also performed the workaround on your server at this time to start the standard EA-NGINX process:

[18:41:26 server2 root@94427991 ~]cPs# systemctl stop nginx-debug
[19:14:33 server2 root@94427991 ~]cPs# systemctl disable nginx-debug
Removed /etc/systemd/system/multi-user.target.wants/nginx-debug.service.
[19:14:39 server2 root@94427991 ~]cPs# /scripts/restartsrv_nginx
Waiting for “nginx” to start ……waiting for “nginx” to initialize ………finished.

Service Status
        nginx (nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf) is running as root with PID 55124 (systemd+/proc check method).

Startup Log
        Mar 22 19:14:46 server2. e .com systemd[1]: Starting nginx - high performance web server...
        Mar 22 19:14:46 server2.e .com systemd[1]: Started nginx - high performance web server.

nginx started successfully.

I have filed a case with our developers regarding this with case ID CPANEL-40292.
I also have created the following article for additional information on this: EA-NGINX fails to start on Ubuntu after a server restart

You can follow the article to receive a notification when an update is available on this as well.

Workaround
Disable and stop the nginx-debug service:

systemctl stop nginx-debug
systemctl disable nginx-debug

Start the proper EA-NGINX process with the following:

/scripts/restartsrv_nginx

Thanks To David again for helping resolve the issue.

 

[cPRex]

Thanks for posting!