Sending Bulk Email

I have a new product that I want to notify my customers of and since it has been a while since I last sent an email blast and I have moved to a new server, I wanted to make sure I was conforming to the latest best practices. And when they sign up I want to make sure that confirmation and billing emails get delivered.

Return Path
The first thing I noticed is that my Return-path and envelope-from were set to the server name rather than the domain of the user that I was sending the email from. There is a way to configure Exim4 to automatically use the domain of the user when sending email, but I couldn’t figure out how to do it. I found this article that explains how to change it. However, since all of the mail I send from that server comes from the same domain, I fixed the problem by changing the /etc/mailname file so that it has the sending domain rather than the server’s domain.

I also updated my SSL config with certbot to set up mail.wellgolly.com as part of the SSL configuration. You need to do this separately from the website using sudo certbot certonly --standalone -d mail.wellgolly.com. However, since I changed the sending domain from mail.wellgolly.com to wellgolly.com this didn’t matter in the end for my bulk email but since my confirmation and billing emails still send out from mail.wellgolly.com. (I’m working on figuring out why they do that—the code uses PHP and was written by my guys 25 years ago, so I haven’t quite figured it out yet.)

Then I added a text file to my DNS. This spf file is not supposed to have a HOSTNAME label. The validator couldn’t find the spf file when it had a label but finds it when I removed the label.
v=spf1 mx a ip4:173.255.194.220 ip6:2600:3c00::f03c:91ff:fe70:bc93 include:charter.net include:wellgolly.com include:mail.wellgolly.com -all

Note that you include all of the servers that you use to send mail. In my case I use an ISP to send mail from my computers and wellgolly.com for the bulk email. As I mentioned earlier, I still haven’t figured out why my order confirmation emails use mail.wellgolly.com so I put that in as well. If you send email from your phone, you’d include that server as well.

DKIM
I found these two links that were useful for setting up DKIM with Exim4. myleen and a concise explanation from Mike Pultz.


remote_smtp:
  debug_print = "T: remote_smtp for $local_part@$domain"
  driver = smtp
  dkim_domain = wellgolly.com
  dkim_selector = x
  dkim_private_key = /etc/exim4/dkim.private.key
  dkim_canon = relaxed

The forum post from myleen says that you can use dkim_domain = ${lc:${domain:$h_from:}} but I didn’t try it.

One thing that I forgot was that the my Exim configuration on my machine is not set up in separate files but is in one large config file, exim4.conf.template. To have the changes take effect you need to run sudo update-exim4.conf and then restart Exim.

Notice that the line dkim_selector = x. The x can be anything but when you add the DKIM key you need to use the same label. In my case, x._domainkey.

Generate the public and private keys as described in the article and add two TXT files

x._domainkey.example.com.   TXT v=DKIM1; t=y; k=rsa; p=<public key>
_domainkey.example.com. t=y; o=~;

These are the TXT files I created so that the validator is happy.
TXT_Records

Verification
You can send an email to check-auth@verifier.port25.com and it will respond with cryptic summary of results. This is good if you understand all the terms and how to fix things. I didn’t, so I used Mail Tester to check my mailings. It’s free for the first three but since I was fixing and testing based on their feedback I ponied up the 25€ ($27.50 with conversion fee of .15) and got 25 tests.

Make sure that Exim has permission to read the dkim.private.key file. This caused some validation errors. I don’t know who the user is that is sending the mail, so I couldn’t add them to a group. What I did instead was make everyone able to read the file using chmod o+rx.

Reverse DNS
I was getting this error:


Your IP address 173.255.194.220 is associated with the domain wellgolly.com.
Nevertheless your message appears to be sent from mail.wellgolly.com.
You may want to change the host name of your server to wellgolly.com.

So to fix this I edited /etc/hosts to remove the mail. prefix.

List Unsubscribe
This is recommended for newsletters so I added it. I can’t see it in the mail readers that I use, but it got rid of some negative scoring on the validator so I added it. I think Spark uses it to indicate that the message is a Newsletter.
Unsubscribe

List-Unsubscribe: <mailto: unsubrequests@exampledomain.com?subject=unsubscribe>, <http://www.exampledomain.com/unsubscribe.html>

This is what the mail validator says now and I score 10/10. Unfortunately, that didn’t prevent Gmail from marking my email as spam, but at least it didn’t get rejected entirely.

Authenticated
Misc
SpamAssasin
WellFormatted

I couldn’t track down what the T_SPF_PERMERROR means but it doesn’t seem to have any effect on my score.

One thing that paradoxically did have an effect on my score was a ratio of text to graphics that was too high. I added a paragraph of text and that issue was resolved.

The results from verifier.port25.com are encouraging as well. Unlike Mail Tester, they don’t tell you how to fix the problems or even really what the results mean. They are free however, and now that I know a bit about what everything means, I’ll probably use them for checking things when I make changes.


==========================================================
Summary of Results
==========================================================
SPF check:          pass
"iprev" check:      pass
DKIM check:         pass
SpamAssassin check: ham

==========================================================
Details:
==========================================================

HELO hostname:  wellgolly.com
Source IP:      2600:3c00::f03c:91ff:fe70:bc93
mail-from:      sales-support@wellgolly.com

----------------------------------------------------------
SPF check details:
----------------------------------------------------------
Result:         pass
ID(s) verified: smtp.mailfrom=sales-support@wellgolly.com

DNS record(s):
   wellgolly.com. 300 IN TXT "v=spf1 mx a ip4:173.255.194.220 ip6:2600:3c00::f03c:91ff:fe70:bc93 include:charter.net include:wellgolly.com include:mail.wellgolly.com-all"
   wellgolly.com. 300 IN MX 1 mail.wellgolly.com.
   mail.wellgolly.com. 300 IN AAAA 2600:3c00::f03c:91ff:fe70:bc93

----------------------------------------------------------
"iprev" check details:
----------------------------------------------------------
Result:         pass (matches wellgolly.com)
ID(s) verified: policy.iprev="2600:3c00::f03c:91ff:fe70:bc93"

DNS record(s):
   3.9.c.b.0.7.e.f.f.f.1.9.c.3.0.f.0.0.0.0.0.0.0.0.0.0.c.3.0.0.6.2.ip6.arpa. 300 IN PTR wellgolly.com.
   wellgolly.com. 300 IN AAAA 2600:3c00::f03c:91ff:fe70:bc93

----------------------------------------------------------
DKIM check details:
----------------------------------------------------------
Result:         pass (matches From: sales-support@wellgolly.com)
ID(s) verified: header.d=wellgolly.com

The simple verifier has some options that you can find at their website. I also used it to check what’s happening with my other mail by sending a command-line email.


/usr/sbin/exim -v check-auth@verifier.port25.com
From: sales-support@wellgolly.com
To: check-auth@verifier.port25.com
Subject: DKIM Test
test message

In case you’ve forgotten, when you send a multi-line command you terminate it with ctl-d.

The Result
I sent out my mass mailing to customers today and, unlike previous years, I did not get any bounces for missing SPF record, line length exceeded, or spam filters. So the couple of days figuring this out was worth it.

I did get a lot of bounces since many people do not keep the same address forever like I do. Many of them are from schools and companies so it makes sense that the email address is deactivated after a while when employees leave. Lots of bounces for mailbox full so most of them are also probably old email addresses.

I dumped all of the undeliverable mail into a file so I can make a list of bad addresses and the first one in the list also has a bunch of diagnostic stuff in it. An interesting bit was this section:


X-Barracuda-Spam-Score: 0.62
X-Barracuda-Spam-Status: No, SCORE=0.62 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=5.0 tests=ANY_BOUNCE_MESSAGE, BOUNCE_MESSAGE, BSF_SC0_SA074b, BSF_SC0_SA590, EMPTY_ENV_FROM, NO_REAL_NAME, SH_BIG5_05413_BODY_104
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.83021
  Rule breakdown below
   pts rule name              description
  ---- ---------------------- --------------------------------------------------
  0.00 EMPTY_ENV_FROM         Empty Envelope From Address
  0.00 NO_REAL_NAME           From: does not include a real name
  0.21 SH_BIG5_05413_BODY_104 BODY: Body: contain "UNSUBSCRIBE"
  0.20 BSF_SC0_SA590          Custom Rule SA590
  0.20 BSF_SC0_SA074b         Custom Rule SA074b
  0.00 BOUNCE_MESSAGE         MTA bounce message
  0.00 ANY_BOUNCE_MESSAGE     Message is some kind of bounce message

According to the Barracuda website my score if .62 is great since The score ranges from 0 (definitely not spam) to 10 or higher (definitely spam). That program is fairly popular and there were 52 other messages that I got back 35 had a score of 0, 6 had a score of 1.09, two had a score of 2.02 and the rest were between .21 and .91. I’m not sure I can do anything about the high scores since the report doesn’t make a lot of sense to me. My embedded URLs use HTTPS and are really short—just the domain name followed by exercises/overview.html. Most of the points come from their custom rules so there’s not much I can do about that.


  Rule breakdown below
   pts rule name              description
  ---- ---------------------- --------------------------------------------------
  0.00 NORMAL_HTTP_TO_IP      Uses a dotted-decimal IP address in URL
  0.00 NO_REAL_NAME           From: does not include a real name
  0.00 MIME_BOUND_MANY_HEX    Spam tool pattern in MIME boundary
  0.00 EMPTY_ENV_FROM         Empty Envelope From Address
  0.32 URI_HEX                URI: URI hostname has long hexadecimal sequence
  0.00 IP_LINK_PLUS           URI: Dotted-decimal IP address followed by CGI
  0.50 WEIRD_PORT             URI: Uses non-standard port number for HTTP
  0.00 HTML_MESSAGE           BODY: HTML included in message
  0.20 BSF_SC0_SA590          Custom Rule SA590
  0.50 BSF_RULE7568M          Custom Rule 7568M
  0.50 BSF_RULE_7582B         Custom Rule 7582B
  0.00 BOUNCE_MESSAGE         MTA bounce message
  0.00 ANY_BOUNCE_MESSAGE     Message is some kind of bounce message>/code>
 

There were only five messages checked with SpamAssassin and this was the result:
<code>
X-Spam-Status: No, score=0.0 required=9.9 tests=HTML_MESSAGE
    autolearn=disabled version=3.3.2
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06)

One message had this report which makes no sense to me, but maybe it will be useful to someone else.


X-ENA-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=3.554,
  required 4, BAYES_00 -2.20, DATE_IN_PAST_06_12 1.54,
  DKIM_SIGNED 0.10, DKIM_VALID -0.10, DKIM_VALID_AU -0.10,
  DKIM_VALID_EF -0.10, DMARC_PASS -0.00, ENA_BAD_OPTOUT 2.20,
  ENA_BAD_OPTOUT5 0.00, ENA_BAYES_OFFSET 2.20, HTML_MESSAGE 0.00,
        ENA_BAD_OPTOUT5 0.00, ENA_BAYES_OFFSET 2.20, HTML_MESSAGE 0.00,
  SPF_HELO_PASS -0.20, SPF_PERMERROR 0.20, T_SPF_PERMERROR 0.01)

There were 36 of these:


X-Forefront-Antispam-Report:
  CIP:134.197.10.234;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:UC-Exchange1.unr.edu;PTR:InfoDomainNonexistent;CAT:NONE;SFTY:;SFS:(50650200002)(4636009)(136003)(396003)(346002)(376002)(39860400002)(1930700014)(46966005)(30864003)(78352004)(42882007)(336012)(26005)(7696005)(498600001)(45080400002)(2876002)(31686004)(8936002)(8676002)(2906002)(83380400001)(6916009)(55016002)(786003)(316002)(31696002)(82310400002)(70586007)(70206006)(66576008)(82740400003)(956004)(47076004)(81166007)(5660300002)(356005)(53652003)(559001)(579004)(299355004);DIR:OUT;SFP:1501;

You can try to decipher it at the Microsoft website but I think the good part for me is CAT:NONE; since all of the listed categories are bad and NONE appears to be the default that indicates that the message is not spam.

One of the things that you need to do to keep off the spam lists is remove old email addresses. I don’t plan on sending bulk emails to my customers very often but if I do I want to stay off the spammer list. So I created a list of bad addresses. The first thing I did was to search for X-Failed-Recipients:. This provided me with a clean list of 86% of the bounces.

I then looked for To: ". This gave me more lines than bounced messages but by filtering out every line that didn’t end in >, sorting the remainder and removing duplicate lines, I got it down to one line per address. 94% of the bounces.

Combining the two gives me slightly fewer addresses than using the To: " method alone. This probably happened because I had people in the database multiple times with slightly different name fields.

There were a handful of Unknown address error messages that didn’t fit any pattern for automatic filtering. They were only 1% of the total so I cleaned them up by hand. Nothing else popped out to me so I think I’ll leave it at that.

git Commands I Use

I started using git with my first Apple app because it was built into Xcode and one of the first lessons in the Stanford Xcode class recommended that we use it. I never used the branching features, since I’m the only one working on the code, but I frequently made use of the lookback features so I could roll back code that didn’t work or grab code that I had discarded but found out that I needed. So I basically used it for journaling.

Most of the websites that I work on don’t have lots of active users so when I want to make an update or something breaks, I just work on a copy or on the live files. However, I recently started a website that has active users and rather than only working in the site at night or making sure no one was using it before potentially breaking it, I decided to make a beta subdomain as I explained in some recent PHP-related posts. As long as I don’t mess with the database I can break whatever I want and it doesn’t affect users.

At first I thought that I’d use rsync to sync the two but git seems to work fine. The only problem is if I fix a small bug on the live site then I have trouble getting the new stuff to synchronize. I suppose I could figure out how to resolve differences, but an easier way is just to force the live site to match the beta site. Since they are on the same server, it doesn’t take any time to synchronize.


 git fetch --all
 git reset --hard origin/master

The normal way to sync when I haven’t made any changes is just:


 git pull origin master

I also keep a copy as backup on my laptop with the same code.


  // Do this once to establish the origin
  git remote add origin "ssh://g@wellgolly.com/www/beta/exercises/.git"
  // Do this when you want to synchronize
  git pull origin master

PHP include paths

One of the nice things about PHP is that it is easy to write one common file for things like headers and footers and then include that file in every page that you publish. For simple sites, it is easy, you just include a line like this in as the first line your code.


<?php
require_once('header.inc'); ?>

where the file is something like this:

<!doctype html>
<html lang="en-US">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="copyright" content="Copyright ©<?php echo $now; ?> Well Golly.com" />
  <meta name="rating" content="general" />
  <meta name="robots" content="all" />
  <meta name="description" lang="en" content="<?php echo $meta_description; ?>"/>
  <meta name="keywords" content="<?php echo $meta_keywords; ?>" />
  <meta name="robots" lang="en" content="index follow" />

    <link rel="icon" type="image/png" href="./common_images/favicon.png" />
    <link rel="apple-touch-icon" href="./common_images/apple-touch-icon.png"/>
    <link rel="stylesheet" href="/css/pure-min.css">
    <title><?php echo $page_title; ?></title>
</head>

Once you have the basic file in place, you can start embellishing it. Note that I refer to several PHP variables. If you examine the source for this page, you see that it is copied literally into the HTML source and a few variables that I previously defined take on their values. These are defined in a file that I call set_variables.inc.


<?php
    $day = date("Y-m-d",strtotime("now"));
    $today = date("Y-m-d");
    $nowDate = date('Y');    // Used for copyright notices in the header tag and in the footer.

  // Here is where you provide information about your site.
  $site_name = "Well Golly";
  $site_shortName = "Well Golly";
  $site_URL = "https://Well Golly.com";

  $copyright_text = "© Copyright $nowDate $site_name";
  $footer_text = "&copy; Copyright $nowDate <a href=\"{$site_URL}\">$site_name</a> $site_address&nbsp;&nbsp;$site_phone";

  $keywords = "$site_name"; // Separate terms with commas
  $description = "$site_name web-based exercises.";

  $meta_description = 'Well Golly Exercises.';
    $meta_keywords = 'products, apps, software';

  $page_title = "$site_name Exercises";

?>

Note that this file starts with because none of the information on the page is HTML. This file defines variables that I use in other files to generate HTML.

One thing that is confusing at first is how PHP finds the included files. If you just use the file name in the call, then PHP looks in the current directory for the file. PHP uses Unix conventions, so if you are in a subdirectory and want to call a file one level above, you would preface the file name with ../ e.g.


<?php
require_once('../header.inc'); ?>

Likewise, two levels up is ../../ and two levels up then down a level is ../../directory. This can get confusing after a while so what most people do is create a directory in the document root and put include files that are used throughout the site in it. Here’s an example of my include.php file for a site.


agreement.inc
check_login.inc
footer_menu.inc
footer.inc
header_logo.inc
header_menu.inc
header.inc
login_validation.inc
set_variables.inc
signup_confirmation.inc
signup_form.inc
signup_updateDB.inc
signup_validation.inc
timeZones.inc

Remembering to put the correct number of ../s can be tiresome so what I do is define a search path and put it in my php.ini file. I don’t remember precisely what it looks like, but this line from my /var/log/php_error.log file tells me where PHP was looking for a file that it can’t find.


 PHP Fatal error:  require_once(): Failed opening required './include.php/header_menu.inc' (include_path='.:/usr/share/php5:/usr/share/php:./include.php:..:../include.php:../../include.php:../../../include.php')

I basically says to look first in the current directory . , then in the PHP provided directories, then in the next level up, etc. If you want to override the search path you can just specify where PHP should look for the file. For example, in the page where users interact with your app, you may not want all of the normal branding and menu choices. Create a simple header file and access it directly by specifying the path.

<?php
require_once('./header.inc'); ?>

You can do the same thing for CSS and javascript. I do this for all of my apps since there is some common javascript but much of it is specific to an app and I don”t want the code to get confused because I used a function name that has a different input and output in other apps. Plus there is less code being downloaded so the page loads faster.

Where it gets tricky is when you start thinking that PHP starts looking in the document root for files and images when in fact it starts looking in the root of the server as defined in your Apache2 conf file. In my case it is /srv/www.

So if you want to include an image in your logo, you might put it in /common_images in the document root. However, if you call it using <img src='/common_images/logo.png' alt=logo' /> PHP won’t find it because it will be looking in /srv/www//common_images/logo.png. Remove the first slash and you are good to go—if you are calling the images from the same level as common_images. You could add the path to your php.ini file, but I have a bunch of folders that I use where I would have to do that. An easier way is to use PHP server variables. In the example below, I have already defined the $site_URL in my set_varialbles.inc file and use a built-in PHP function, explode, to get the filename of the directory where my common_images directory is located. So no matter how far down the directory tree I am, I can still get the location where the logo is located—https://beta.wellgolly.com/exercises/common_images.


<div id="header" class="pure-g">
  <div class="pure-u-1 pure-u-lg-1-2">
    <a href="<?php echo $site_URL ?>">
    <?php 
      $curRoot = explode("/", $_SERVER['REQUEST_URI']);
    ?>
      <img class="pure-img-responsive" src="<?php echo "{$site_URL}/{$curRoot[1]}" ?>/common_images/wg_header.png" alt="Header">
    </a>
  </div>
</div>

Finding paths with PHP

I’m worling on web apps and need to do two things that require knowing where you are in the file hierarchy. When a user clicks on an app that I am charging for, I need them to log in. But when they are done with the login, I want to send them back to the app that they want to use. So what I do is use the server variable PHP_SELF to get the complete URL of the file they clicked on. I need to make sure that there is a active session and then add a session variable, calledFileName. Once the user has finished the login or signup process, I redirect them to the app they want to use.


// if user is not logged in, restrict what they can do
$calledFileName = $_SERVER['PHP_SELF'];
session_start();
$_SESSION['calledFileName'] = $calledFileName;

When they hit the submit button, I call a bunch of validation code and reload the form page. Before any HTML is sent, I check to see how they got to the login/signup page and then send them to the appropriate page.


$startlocation = $_SESSION['calledFileName'];
if ($passwordIsValid) {
  if ( strlen($startlocation) > 0 ) {
    header("Location: $startlocation");
  } else {
    header("Location: /exercises/overview.php");
  }
}

Most of my web pages have been fairly simple so I the past I have done live editing to fix mistakes or update text. When I converted one of my sites to responsive HTML5, I created a beta subdomain, e.g. beta.wellgolly.com, and did all the changes there and them rsynced the whole thing to the original site. That worked because I used hard coded paths and one global variable for the site domain name.

I have active users of the exercises and I don’t want to break things for them so live editing is out. I am developing on a beta subdomain, but I want to be extra careful that nothing breaks. I also want to make the site robust enough that I can easily move it to another domain if someone wants to license the apps. So rather than absolute paths, I’ve been using relative paths. This mostly works, but there are some instances where I want to use the same file in different locations and the images that file calls need a fixed path.

I have been using git to sync the files and it seems to work as well as rsync. Because I want to be extra careful about breaking the site for active users, I created a test folder for the exercises on the live site, exercises_test, and synched it with the beta content. Then renamed it when I was sure everything was working. The relative paths work fine but hard coded paths won’t work in this case since the files are not located in /exercises/folder but are located in /exercises_test/folder. One case where it doesn’t work is when I redirect the user to the login page if they haven’t yet logged in. In this case, I use another server variable and grab the root directory. The variable looks like this: /exercises_test/apptype/index.php so when I grab the root folder with explode, the first element of the array is empty, so I need to get the second element of the array.


if ($_SESSION['userID'] == '') {
  header("HTTP/1.1 302 Found");
  $curRoot = explode("/", $_SERVER['REQUEST_URI']);
    header("Location: /$curRoot[1]/login.php");
    die();
}

I can use this same trick for path names in my header files.

PHP password storage in MariaDB

Years ago I built a site that had login and account creation capability using SHA1 and MySQL. In preparation for a new webiste I loaded the old site into a droplet it still works fine. At the time SHA1 and simple input sanitizing was fine, but since then SHA1 has been cracked and PDO is recommended for sanitizing database values.

The original code looked like this:


// Read the data from the form
$userlogin = htmlspecialchars($_POST['login']);
$password  = htmlspecialchars($_POST['password'];
// The hash stored in the database.
$passwdHash = sha1($Password);

// Verification
$userlogin = htmlspecialchars($_POST['login']);
$password  = htmlspecialchars$_POST['password'];
$passwdHash = sha1($password);
$sql = "SELECT *
FROM users
WHERE username = '$login' && password = '$passwdHash' ";

Without really thinking about it, I sanitized the password input with htmlspecialchars and in both storing and retrieving the password it didn’t have any impact.

I basically copied the code into my new project and then changed SHA1 to password_hash. That didn’t work because the password_hash requires password_verify to check the hash when you retrieve it from the database. That worked when I tested the login code by copying and pasting the login info from php_error.log –> error_log(login: $userlogin  hash: $hash) into the users table. It didn’t work when I created a signup page and entered the data automatically. The reason is that sanitizing the password with htmlspecialchars changes it somehow. This is what I do instead. Note that it doesn’t matter what the user enters in the password field because the hash function transforms it into a 60 character hash so there is no need to sanitize.


// Read the data from the form
// Sanitized passord is used in javascript just for checking that a password was entered
$Password = htmlspecialchars($_POST["Password"]); 
$UserLogin = htmlspecialchars($_POST["UserLogin"]);
$PasswdHash = password_hash($_POST["Password"], PASSWORD_DEFAULT);

// Enter into the database with PDO
$sql = "INSERT INTO `Database`.`users` (`user_login`, `user_pass`,
$sql .=  "VALUES (:user_login, user_pass, ...

$stmt->bindParam(': user_login', $UserLogin);
$stmt->bindParam(':user_pass',   $PasswdHash);
...

// Validate the password. Retrieve by user_login only. 
// Don’t do anything to the password that was entered by the user.

foreach($result as $row) {
  $user_login  = $row['user_login'];
  $hash = $row['user_pass'];

  if ( password_verify($_POST['password'], $hash) ) {
    $_SESSION['userID']    = $row['ID'];
    $_SESSION['userName']   = $row['user_name'];
    $_SESSION['userDisplayName'] = $row['display_name'];
    $passwordIsValid = TRUE;
  }