Archive for the ‘Coding’ Category

Understanding User Requests

Tuesday, September 11th, 2018

bobwaycott at ycombinator made a comment that summarizes my thoughts on developing software for end users.

Always. I’ve been practicing this for 10 years, most of which I’ve been working as a consultant building custom software for internal business processes, as well as customer-facing software for clients’ users. My typical process looks something like this when a feature/change request or random idea is asked for—which almost always comes in along the lines of, “Can we get X added in that does Y and looks like Z?”:

1. I never say yes to any request immediately. I always tell a client, “Anything is possible that doesn’t violate the laws of physics, but let’s dig into this more.”

2. Ask what they’re trying to accomplish. What problem are they trying to solve? What mistake are they trying to prevent? What is the end goal? I ask questions until I can explain back to the customer what they’re really wanting to do and why.

3. I the push back on why I think we should not do what they’re asking for in the way they’re asking for it. When doing this, I always try educating them on the tech behind the scenes, potential pitfalls, how adding something (especially if it’s visible to people) will make it nearly impossible to ever remove it once users get used to it, when they’re asking for something that would be more effort the way they’re asking for it to be done than it’s worth when it’s trying to solve for a rare edge case, etc.
4. After explaining the problems with their idea as given by non-experts, I start suggesting ways we could accomplish their goals with a simpler UX, or even no UX at all, relying on the ability to automate things if we have enough information, or hide all the complexities of a process behind a single button once we have the right info to intelligently take action.

5. I give a couple of recommendations for potential ways forward that solve the real problem in a way I’ll enjoy building it out. Then I let them make the choice.

Following this pattern has pretty much never failed me. What feels best about it is when I see clients actually learn how their software works when I’m working for them. I love it when they remember the discussions we’ve had, internalized it, and recall it when we talk 6 months later about their new idea. Over time, their ideas improve because their understanding of how their software works improves. They also become increasingly invested in our working relationship as their trust in my concern for solving their problems—and not just doing their bidding—increases.

Never shy away from challenging your customers’ ideas—but always do it in a respectful manner that gets to the heart of their real problems and educated them along the way. They’ll appreciate it, and will keep coming to you for more. I don’t think this is unique to being a consultant, either—the same sort of process can be followed with direct users of your own product

Updating Ubuntu LTS

Wednesday, August 22nd, 2018

I have a small site that was running Ubuntu 16.04 LTS and upgraded it to 18.04 with no issues. I have a couple of customizations in my php.ini and the location where I store my email and because I accepted the maintainer’s version of the PHP and Exim4 files, I had to update a few files to get everything working as before. Not a big deal and the whole process took about a half hour.

On my main site, I was running Ubuntu 14.04 LTS and it was a major project to update. There were times that I considered reverting to the snapshot I took just before I started the update process, but I got everything working about the same as it was before.

The first issue I encountered was with the update to the Linux kernal. The instructions recommended that I go to the Linode control panel and change the Boot Settings Kernel to Grub. When I did that, the server wouldn’t reboot. So I picked the Latest 64 bit version instead and rebooted fine.

Because I was on an older version I had to update in stages. Somewhere along the line it messed up my MySQL install. So I removed MySQL and installed MariaDB. That ended up causing a lot of work which I’ll get to later.

Restoring mail service was pretty straightforward—just changing the location of the virtuals files.

After working through the install process I loaded all of my sites and some displayed properly, some displayed without most of the formatting and some displayed a blank page. It took a while to figure out what was going on. WordPress sites weren’t displaying because of database access issues. There was lots of really weird behaviour on the other sites that I eventually tracked down to permissions problems on a shared styles folder. For example, if I accessed a site with sitename.com things looked fine. But if I accessed it with www.sitename.com, I got no formatting. And in one case it redirected to the default site for the server. It took quite a while to figure out what was happening but eventually I tracked it down to a permissions problem. I keep styles that I use on all my sites in a sim-linked folder in /www/common/Styles. After I changed all of the file permissions to -rw-rw-r– and the directory permissions to dr-xr-xr-x, the sites started behaving normally.

The MySQL issues were much more difficult to resolve and took the better part of two days. There were enough changes to the users file that I couldn’t just copy it into /var/lib/mysql. I went with the default install and then took then loaded users (and their database permissions) from my mysqldump from the night before. I took the opportunity to delete users who no longer needed access to the system and clean up some of the permissions on those who remained.

Copying the sql files from a backup copy worked for most tables, however there were some that only had a .frm file and no MYD and MYI file. PhpMyAdmin showed them as being “in use” and would not display their contents. I reloaded their contents from the mysqldump. That solved most of my website display problems, but I couldn’t log into any WordPress installations. That turned out to be a permissions problem. I changed the user and group of all the sql files to mysql and then changed all of the permissions on the files to -rw-rw-r– and folders to drwxr-xr-x. Now all of my sites were displaying properly.

The only problem was with the order confirmation page. I got an error Cannot start session when headers already sent error. I had a redirect on one page that was in fact trying to redirect after some html had already been sent. Not sure why it worked before, but putting in a conditional to not load the html if the page was going to be redirected fixed it.

The order page still didn’t work and the reason was that it used mc_encrypt which is no longer supported in PHP 7.2. I updated the files to use openssl_encrypt and everything works as before. I didn’t write the original order form code so it took a while to understand how the encryption works. There are lots of examples out there, including on the PHP.net site, so I won’t go into detail about what I did. Understanding and implementing the new encryption method probably took around three hours.

An older set of pages that used mysql_real_escape_string also failed to load because mysql_real_escape_string was no longer supported. In this case, replacing it with htmlspecialchars solved the problem and should have been what I used to begin with. This section of the site also had some date returns that did not correctly use the date function. I replaced $thisMonth = date(F); with $thisMonth = date(‘F’); and did the dame thing with $thisDay = date(d);

For some reason, aptitude wasn’t upgraded so I had to install it when I did the first update of the site.

Apache not creating log files.

Saturday, July 16th, 2016

My Apache log files are not set up to be rotated—not sure why my Ubuntu install didn’t do that. But there are dozens of them and rather than clearing them out individually, I decided to delete all of them. After I deleted them, Apache didn’t recreate them. It turns out that Apache creates them when it is restarted so a restart of Apache recreated the files. I like to edit them with BBEdit, so I needed to also change permissions and the group.


sudo chmod -R u+rwX,go+rwX,o-w .
sudo chgrp adm *

Converting another site to https

Thursday, May 26th, 2016

I have another site that is fairly small and uses a responsive design template. I decided to convert that site to use https with certbot which is EFF’s Client for Let’s Encrypt Let’s Encrypt.

I had previously played around with using a self-signed certificate so I went to /etc/ssl and removed all of the keys for the site. Then I followed the directions on the certbot site to let it guide me through the process. I noticed when I removed the old certificates that the directories were owned by root and that I couldn’t cd into the directories without first changing to root. Thinking that certbot would put the new certificates in the same place (which it doesn’t) I decided to run the commands as root. So after downloading certbot-auto and changing the permissions, I changed to root by using ‘sudo su’. The bot is really smart about looking at your system and figuring out which domains you are hosting and which already have certificates. It appears that it looks at the sites-available files and if it finds a section for https (set off with <VirtualHost *:443> ... </VirtualHost> it assumes that there is a certificate already in place. So the domain I was targeting didn’t show up in the list. Removing that section from the file and running certbot again found the site.

Certbot doesn’t use that section so after the installation it added these lines to the sites-available file.


RewriteEngine on
RewriteCond %{SERVER_NAME} =wellgolly.com [OR]
RewriteCond %{SERVER_NAME} =www.wellgolly.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]

I did not have to restart Apache. When I visited the site I noticed that it was not displaying properly. The cause was http links in the header for fonts and bootstrap. Changing all of the links to https fixed the display problems.

The next thing was to look for all of the internal links that needed to be changed to https. I used the link checker at W3.org to check for links that I missed. Finally I went to SSL Labs and tested the site. It got an A+.

Unlike the other certificates which are stored in /etc/ssl, certbot puts all of its stuff in /etc/letsencrypt/live/.

Also, rather than adding the section with the https port (443) to your vhost file, it creates a separate one for https. It uses the same name and appends ‘-le-ssl.conf’.

My site is fairly straightforward, but you may need to add things to this file if you have a more complicated setup.

If you have more than one domain that points to the same site, you can use the same certificate. If www.myreallylongdomainname.com points to www.mydomain.com then after you set up the certificate for one domain, you can use the same certificate for the others. Unlike when setting up the original certificate, you do need to restart Apache for the changes to take effect.


sudo ./certbot-auto certonly --cert-name example.com -d mydomain.com,www.mydomain.com,www.mydomain.com,myreallylongdomainname.com,www.myreallylongdomainname.com
service apache2 restart

IF you have lots of sites on the same server you can see which ones you have certificates for by running the command:


sudo ./certbot-auto certificates

You will get an email when the certificate is set to expire. CD to the directory where certbot-auto is located and run


sudo ./certbot-auto renew

You can also check with an entry to a cron job. I set up one to run monthly. It looks like this:


0 0 1 * * /home/userid/certbot-auto renew -q

Processing Remove Requests

Wednesday, May 25th, 2016

I stumble upon an easier way to filter requests for removal from our email list and add them to our remove table in the database.

It turns out that in Apple Mail, if you select a bunch of emails and then click on the Forward button, it selects the content of each email and pastes it into a new email. For our remove folder, the process took a while, but the end result is something like this:
Remove_Email

There is a whole bunch of stuff I don’t need, but Mail nicely highlights the From name and address.

I selected the text and pasted it into BBEdit. Then I used the process lines feature to extract all lines starting with From: (case sensitive). There were a couple with Well Golly in them, but otherwise it was a pretty clean result, with one address per line, name first and email addresses in <> e.g.


From: DarrleneBarrett@aol.com
From: John Sollino <johnc1459@hotmail.com>
From: "Jane Alexander" <jalexander156@hotmail.com>
From: "Strand, Bill, Ph.D." <Strand.Bill@commoncore.edu>

From here it is easy to massage this into a form that can be imported into the database.

Well Golly


Atheism Plus

Buy from Amazon