All sites down after update

jonathannz · Feb 1, 2023

I have four sites installed on this server. I updated WordOps and the server stack. Out put of that was:

natnut@nhnut:~$ wo update
WordOps changelog available on https://github.com/WordOps/WordOps/releases/tag/v3.16.3
Do you want to continue:[y/N]y
Downloading update script        [Done]
updating WordOps, please wait...

Welcome to WordOps install/update script v3.16.3

Installing wo dependencies	[OK]
Installing WordOps	[OK]
Running post-install steps	[OK]
Synchronizing wo database, please wait...
WordOps (wo) upgrade to v3.16.3 was succesfull!
Changelog is available on https://github.com/WordOps/WordOps/releases/tag/v3.16.3

To upgrade WordOps web stacks, you can use the command:
wo stack upgrade

To update all other packages, you can use the command:
wo maintenance

WordOps Documentation : https://docs.wordops.net
WordOps Community Forum : https://community.wordops.net
WordOps Community Chat : https://chat.wordops.net

Give WordOps a GitHub star : https://github.com/WordOps/WordOps/

natnut@nhnut:~$ wo maintenance
updating apt-cache, please wait...
updating packages, please wait...
cleaning-up packages, please wait...
natnut@nhnut:~$ wo stack upgrade
[sudo] password for natnut:
Nginx Stable is not already installed
Your sites may be down for few seconds if you are upgrading Nginx, PHP-FPM, MariaDB or Redis
Do you want to continue:[y/N]y
Updating APT cache              [OK]
Configuring php7.3-fpmes        [..]
Restarting php7.3-fpm           [OK]
Configuring php7.4-fpm
Restarting php7.4-fpm           [OK]
Tuning MySQL configuration      [OK]
Restarting mariadb              [OK]
Restarting fail2ban             [OK]
Configuring Fail2Ban
Configuring APT Packages        [OK]
Downloading WP-CLI               [Done]
Downloading Netdata              [Done]
Downloading WordOps Dashboard    [Done]
Downloading PHPMyAdmin           [Done]
Downloading Adminer              [Done]
Downloading Adminer theme        [Done]
Downloading Composer             [Done]
Downloading MySQLTuner           [Done]
Downloading ngxblocker           [Done]
Stop  : netdata   [OK]
Upgrading Netdata               [OK]
Upgrading Composer              [OK]
Upgrading phpMyAdmin            [OK]
Successfully updated packages

Nothing in that stands out to me.

In the browser I am getting the ERR_CONNECTION_REFUSED error. So it seems I'm not even getting through to the server stack.

I've tried accessing via a VPN in case my IP had someone got blacklisted by something (?), but that made no difference.

I've tried restarting the stack, and also restarting the server.
I see no errors arising in the site specific error logs. As mentioned, it seems my requests are not even making it through to the stack, hence no site errors are generated.

I am not sure what else to try. Any suggestions would be appreciated.

Thank you

jonathannz · Feb 1, 2023

I issued the stack install command just to see what would happen (was seeing if I could reinstall the existing stack that way). The output I got was ...

natnut@nhnut:~$ wo stack install
PHP 7.4 already installed
MySQL already installed and alive
WP-CLI is already installed
Fail2ban already installed
Sendmail already installed
phpMyAdmin already installed
phpRedisAdmin already installed
Composer already installed
Adminer already installed
MySQLtuner already installed
Netdata already installed
eXtplorer is already installed
Adding repository for NGINX, please wait...
Updating apt-cache              [OK]
Oops Something went wrong!!     [..]
Check the WordOps log for more details `tail /var/log/wo/wordops.log` and please try again...

So I checked the wordops log and got:

natnut@nhnut:~$ tail /var/log/wo/wordops.log
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 nginx-custom : Depends: nginx-common (= 1.22.1-1ppa~stable~ubuntu20.04.1) but 1.23.2-1+ubuntu20.04.1+deb.sury.org+1 is to be installed
 nginx-wo : Depends: nginx-common (= 1.22.1-1ppa~stable~ubuntu20.04.1) but 1.23.2-1+ubuntu20.04.1+deb.sury.org+1 is to be installed
E: Unable to correct problems, you have held broken packages.
2023-02-01 09:50:53,027 (INFO) wo : Oops Something went wrong!!
2023-02-01 09:50:53,028 (ERROR) wo : Check the WordOps log for more details `tail /var/log/wo/wordops.log` and please try again...

Is it possible those dependency issues would bring the stack down?

jonathannz · Feb 1, 2023

I tried to test the Nginx configuration, only to discover Nginx is no longer installed. Seems the WO update blitzed it.

natnut@nhnut:~$ nginx -t

Command 'nginx' not found, but can be installed with:
 ...

So I tried to perform wo stack install --nginx the that same dependency error above.

Any suggestions on how to get around the dependancy error? Can I just manually install Nginx myself, or will that screw things up in WO?

jonathannz · Feb 1, 2023

I realised I went through this same issue back in Nov 2021. Same server.

See here: WordOps/WordOps#407

How can I get some feedback on how to get the Nginx install back on track?

jonathannz · Feb 1, 2023

For now, I've got around it by downgrading nginx-common to the version that WO requires.

I'd be very interested to know what WO requires an older version that what the system has.

sudo apt-get install nginx-common=1.22.1-1ppa~stable~ubuntu20.04.1

Ttekniboss · Feb 1, 2023

By any chance, are you on a KVM or Openvz VPS?

jonathannz · Feb 1, 2023

tekniboss Hi there. No, it's a regular VM, with MS Azure.

jonathannz · Feb 19, 2023

Today I did a wo stack update command, and it killed nginx again. Nginx in effect gets uninstalled due to the same dependancy issue.

The following packages have unmet dependencies:
 nginx-custom : Depends: nginx-common (= 1.22.1-1ppa~stable~ubuntu20.04.1) but 1.23.3-1+ubuntu20.04.1+deb.sury.org+1 is to be installed
 nginx-wo : Depends: nginx-common (= 1.22.1-1ppa~stable~ubuntu20.04.1) but 1.23.3-1+ubuntu20.04.1+deb.sury.org+1 is to be installed
E: Unable to correct problems, you have held broken packages.

I'd really appreciate some assistance in getting this server back to a correct configuration again. It has numerous sites on it, and having them all go down whenever the server is maintained is a massive issue.

Anyone? @VirtuBox ?

Getting desperate here. The whole reason I use WordOps was to simplify the site maintenance and avoid having to get into the technical aspects of it too deeply. But it seems to be having the opposite outcome.

VirtuBox · Feb 25, 2023

jonathannz Hello,
it's look like you added the following PPA : ppa:ondrej/nginx-mainline
Please try the following command : sudo add-apt-repository -r ppa:ondrej/nginx-mainline
Then run sudo apt update and try to run wo stack install --nginx or wo stack upgrade

jonathannz · Jan 7, 2024

Hi @"VirtuBox" I am attempting to resolve this issue again. Previously I downgraded nginx-common as a workaround, as removing ppa:ondrej/nginx-mainline did not resolve the issue.

Today I have upgraded WO again, and same issue occurred. NGINX gets uninstalled due to a dependancy issue. As shown here:

distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 nginx-custom : Depends: nginx-common (= 1.24.0-1ppa~stable~ubuntu20.04.1) but 1.25.3-1+ubuntu20.04.1+deb.sury.org+1 is to be installed
 nginx-wo : Depends: nginx-common (= 1.24.0-1ppa~stable~ubuntu20.04.1) but 1.25.3-1+ubuntu20.04.1+deb.sury.org+1 is to be installed
E: Unable to correct problems, you have held broken packages.
2024-01-07 18:30:03,064 (INFO) wo : Oops Something went wrong!!
2024-01-07 18:30:03,064 (ERROR) wo : Check the WordOps log for more details `tail /var/log/wo/wordops.log` and please try again...

I have again had to manually downgrade nginx-common, as shown here:

natnut@nhnut:~$ sudo aptitude install nginx-wo
The following NEW packages will be installed:
  nginx-wo{b}
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 21.7 kB of archives. After unpacking 34.8 kB will be used.
The following packages have unmet dependencies:
 nginx-wo : Depends: nginx-common (= 1.24.0-1ppa~stable~ubuntu20.04.1) but 1.25.3-1+ubuntu20.04.1+deb.sury.org+1 is installed
The following actions will resolve these dependencies:

     Keep the following packages at their current version:
1)     nginx-wo [Not Installed]

Accept this solution? [Y/n/q/?] n
The following actions will resolve these dependencies:

     Downgrade the following packages:
1)     nginx-common [1.25.3-1+ubuntu20.04.1+deb.sury.org+1 (now) -> 1.24.0-1ppa~stable~ubuntu20.04.1 (focal)]

Accept this solution? [Y/n/q/?] y
The following packages will be DOWNGRADED:
  nginx-common
The following NEW packages will be installed:
  nginx-wo
0 packages upgraded, 1 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 57.0 kB of archives. After unpacking 112 kB will be freed.
Do you want to continue? [Y/n/?] y
Get: 1 http://ppa.launchpad.net/wordops/nginx-wo/ubuntu focal/main amd64 nginx-common all 1.24.0-1ppa~stable~ubuntu20.04.1 [35.3 kB]
Get: 2 http://ppa.launchpad.net/wordops/nginx-wo/ubuntu focal/main amd64 nginx-wo amd64 1.24.0-1ppa~stable~ubuntu20.04.1 [21.7 kB]
Fetched 57.0 kB in 1s (41.2 kB/s)
dpkg: warning: downgrading nginx-common from 1.25.3-1+ubuntu20.04.1+deb.sury.org+1 to 1.24.0-1ppa~stable~ubuntu20.04.1
(Reading database ... 146070 files and directories currently installed.)
Preparing to unpack .../nginx-common_1.24.0-1ppa~stable~ubuntu20.04.1_all.deb ...
Unpacking nginx-common (1.24.0-1ppa~stable~ubuntu20.04.1) over (1.25.3-1+ubuntu20.04.1+deb.sury.org+1) ...
Selecting previously unselected package nginx-wo.
Preparing to unpack .../nginx-wo_1.24.0-1ppa~stable~ubuntu20.04.1_amd64.deb ...
Unpacking nginx-wo (1.24.0-1ppa~stable~ubuntu20.04.1) ...
Setting up nginx-common (1.24.0-1ppa~stable~ubuntu20.04.1) ...
Installing new version of config file /etc/init.d/nginx ...
Installing new version of config file /etc/logrotate.d/nginx ...
Installing new version of config file /etc/nginx/fastcgi.conf ...
Installing new version of config file /etc/nginx/fastcgi_params ...
Installing new version of config file /etc/nginx/mime.types ...

Configuration file '/etc/nginx/nginx.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** nginx.conf (Y/I/N/O/D/Z) [default=N] ? N
Installing new version of config file /etc/nginx/scgi_params ...

Configuration file '/etc/nginx/sites-available/default'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** default (Y/I/N/O/D/Z) [default=N] ? N
Installing new version of config file /etc/nginx/snippets/fastcgi-php.conf ...
Installing new version of config file /etc/nginx/uwsgi_params ...
Setting up nginx-wo (1.24.0-1ppa~stable~ubuntu20.04.1) ...
Processing triggers for systemd (245.4-4ubuntu3.22) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for ufw (0.36-6ubuntu1.1) ...

But I don't want to have to keep doing this every time I update WordOps.

So you have any other suggestions? I'd really like to resolve this, as it has been going on for a couple of years.