Updating Ubuntu LTS

I have a small site that was running Ubuntu 16.04 LTS and upgraded it to 18.04 with no issues. I have a couple of customizations in my php.ini and the location where I store my email and because I accepted the maintainer’s version of the PHP and Exim4 files, I had to update a few files to get everything working as before. Not a big deal and the whole process took about a half hour.

On my main site, I was running Ubuntu 14.04 LTS and it was a major project to update. There were times that I considered reverting to the snapshot I took just before I started the update process, but I got everything working about the same as it was before.

The first issue I encountered was with the update to the Linux kernal. The instructions recommended that I go to the Linode control panel and change the Boot Settings Kernel to Grub. When I did that, the server wouldn’t reboot. So I picked the Latest 64 bit version instead and rebooted fine.

Because I was on an older version I had to update in stages. Somewhere along the line it messed up my MySQL install. So I removed MySQL and installed MariaDB. That ended up causing a lot of work which I’ll get to later.

Restoring mail service was pretty straightforward—just changing the location of the virtuals files.

After working through the install process I loaded all of my sites and some displayed properly, some displayed without most of the formatting and some displayed a blank page. It took a while to figure out what was going on. WordPress sites weren’t displaying because of database access issues. There was lots of really weird behaviour on the other sites that I eventually tracked down to permissions problems on a shared styles folder. For example, if I accessed a site with sitename.com things looked fine. But if I accessed it with www.sitename.com, I got no formatting. And in one case it redirected to the default site for the server. It took quite a while to figure out what was happening but eventually I tracked it down to a permissions problem. I keep styles that I use on all my sites in a sim-linked folder in /www/common/Styles. After I changed all of the file permissions to -rw-rw-r– and the directory permissions to dr-xr-xr-x, the sites started behaving normally.

The MySQL issues were much more difficult to resolve and took the better part of two days. There were enough changes to the users file that I couldn’t just copy it into /var/lib/mysql. I went with the default install and then took then loaded users (and their database permissions) from my mysqldump from the night before. I took the opportunity to delete users who no longer needed access to the system and clean up some of the permissions on those who remained.

Copying the sql files from a backup copy worked for most tables, however there were some that only had a .frm file and no MYD and MYI file. PhpMyAdmin showed them as being “in use” and would not display their contents. I reloaded their contents from the mysqldump. That solved most of my website display problems, but I couldn’t log into any WordPress installations. That turned out to be a permissions problem. I changed the user and group of all the sql files to mysql and then changed all of the permissions on the files to -rw-rw-r– and folders to drwxr-xr-x. Now all of my sites were displaying properly.

The only problem was with the order confirmation page. I got an error Cannot start session when headers already sent error. I had a redirect on one page that was in fact trying to redirect after some html had already been sent. Not sure why it worked before, but putting in a conditional to not load the html if the page was going to be redirected fixed it.

The order page still didn’t work and the reason was that it used mc_encrypt which is no longer supported in PHP 7.2. I updated the files to use openssl_encrypt and everything works as before. I didn’t write the original order form code so it took a while to understand how the encryption works. There are lots of examples out there, including on the PHP.net site, so I won’t go into detail about what I did. Understanding and implementing the new encryption method probably took around three hours.

An older set of pages that used mysql_real_escape_string also failed to load because mysql_real_escape_string was no longer supported. In this case, replacing it with htmlspecialchars solved the problem and should have been what I used to begin with. This section of the site also had some date returns that did not correctly use the date function. I replaced $thisMonth = date(F); with $thisMonth = date(‘F’); and did the dame thing with $thisDay = date(d);

For some reason, aptitude wasn’t upgraded so I had to install it when I did the first update of the site.

Apache not creating log files.

My Apache log files are not set up to be rotated—not sure why my Ubuntu install didn’t do that. But there are dozens of them and rather than clearing them out individually, I decided to delete all of them. After I deleted them, Apache didn’t recreate them. It turns out that Apache creates them when it is restarted so a restart of Apache recreated the files. I like to edit them with BBEdit, so I needed to also change permissions and the group.


sudo chmod -R u+rwX,go+rwX,o-w .
sudo chgrp adm *

Converting another site to https

I have another site that is fairly small and uses a responsive design template. I decided to convert that site to use https with certbot which is EFF’s Client for Let’s Encrypt Let’s Encrypt.

I had previously played around with using a self-signed certificate so I went to /etc/ssl and removed all of the keys for the site. Then I followed the directions on the certbot site to let it guide me through the process. I noticed when I removed the old certificates that the directories were owned by root and that I couldn’t cd into the directories without first changing to root. Thinking that certbot would put the new certificates in the same place (which it doesn’t) I decided to run the commands as root. So after downloading certbot-auto and changing the permissions, I changed to root by using ‘sudo su’. The bot is really smart about looking at your system and figuring out which domains you are hosting and which already have certificates. It appears that it looks at the sites-available files and if it finds a section for https (set off with <VirtualHost *:443> ... </VirtualHost> it assumes that there is a certificate already in place. So the domain I was targeting didn’t show up in the list. Removing that section from the file and running certbot again found the site.

Certbot doesn’t use that section so after the installation it added these lines to the sites-available file.


RewriteEngine on
RewriteCond %{SERVER_NAME} =wellgolly.com [OR]
RewriteCond %{SERVER_NAME} =www.wellgolly.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]

I did not have to restart Apache. When I visited the site I noticed that it was not displaying properly. The cause was http links in the header for fonts and bootstrap. Changing all of the links to https fixed the display problems.

The next thing was to look for all of the internal links that needed to be changed to https. I used the link checker at W3.org to check for links that I missed. Finally I went to SSL Labs and tested the site. It got an A+.

Unlike the other certificates which are stored in /etc/ssl, certbot puts all of its stuff in /etc/letsencrypt/live/.

Also, rather than adding the section with the https port (443) to your vhost file, it creates a separate one for https. It uses the same name and appends ‘-le-ssl.conf’.

My site is fairly straightforward, but you may need to add things to this file if you have a more complicated setup.

If you have more than one domain that points to the same site, you can use the same certificate. If www.myreallylongdomainname.com points to www.mydomain.com then after you set up the certificate for one domain, you can use the same certificate for the others. Unlike when setting up the original certificate, you do need to restart Apache for the changes to take effect.


sudo ./certbot-auto certonly --cert-name mydomain.com -d mail.mydomain.com,www.mydomain.com,www.mydomain.com,myreallylongdomainname.com,www.myreallylongdomainname.com
service apache2 restart

IF you have lots of sites on the same server you can see which ones you have certificates for by running the command:


sudo ./certbot-auto certificates

You will get an email when the certificate is set to expire. CD to the directory where certbot-auto is located and run


sudo ./certbot-auto renew

You can also check with an entry to a cron job. I set up one to run monthly. It looks like this:


0 0 1 * * /home/userid/certbot-auto renew -q

Processing Remove Requests

I stumble upon an easier way to filter requests for removal from our email list and add them to our remove table in the database.

It turns out that in Apple Mail, if you select a bunch of emails and then click on the Forward button, it selects the content of each email and pastes it into a new email. For our remove folder, the process took a while, but the end result is something like this:
Remove_Email

There is a whole bunch of stuff I don’t need, but Mail nicely highlights the From name and address.

I selected the text and pasted it into BBEdit. Then I used the process lines feature to extract all lines starting with From: (case sensitive). There were a couple with Well Golly in them, but otherwise it was a pretty clean result, with one address per line, name first and email addresses in <> e.g.


From: DarrleneBarrett@aol.com
From: John Sollino <johnc1459@hotmail.com>
From: "Jane Alexander" <jalexander156@hotmail.com>
From: "Strand, Bill, Ph.D." <Strand.Bill@commoncore.edu>

From here it is easy to massage this into a form that can be imported into the database.

PHP Warning

I’ve been getting this warning when I open PhpMyAdmin and since it doesn’t affect anything, I haven’t bothered to fix it.


PHP Warning:  is_readable(): open_basedir restriction in effect. File(js/codemirror/lib/codemirror.js) is not within the allowed path(s): (/usr/share/phpmyadmin/:/etc/phpmyadmin/:/var/lib/phpmyadmin/) in /usr/share/phpmyadmin/js/get_scripts.js.php on line 31

Today I had some time, so I looked at what might be causing it. By grepping on part of the error message,


/usr/share/phpmyadmin/:/etc/phpmyadmin/

I found four files with that text, and more importantly, it was prefixed with
‘php_admin_value open_basedir’.
They were:

./apache2/conf.d/phpmyadmin.conf
./apache2/conf-enabled/phpmyadmin.conf
./apache2/sites-available/wellgolly.com
./apache2/sites-enabled/wellgolly.com

I tried adding ‘/usr/share/javascript/’ to the first one and restarting Apache, but that did’t work. Then I added it to the sites-available file and the did the trick.