Author: JS

I’ve been doing a lot of text manipulation lately. Here are some tricks that I don’t want to forget.

I’m working on an FAA Glossaries iPod app. I got the words from FAA publications. They are in a MySQL database that I export to SQLite for use in the app. Here are some tricks I’ve been using to clean up the data before and after import to the database.

grep for line numbers

After exporting from the database I have a file that starts with a parenthesis a number, a comma, and a space.
The following grep code will remove the parenthesis, one or more numbers, the comma, and the space (indicated by a b).

Finding duplicate occurrences of a set of characters in a line.

The original PDFs and web pages are fairly consistent so it’s not too difficult to automate the process of converting a glossary to a format that I can import into the database. Eventually I want it to look like this:

(2983, ‘advection’, ‘The horizontal transport of air or atmospheric properties. In meteorology, sometimes referred to as the horizontal component of convection.’, 7, 4),
(2984, ‘advection fog’, ‘Fog resulting from the transport of warm, humid air over a cold surface.’, 7, 4),
(2985, ‘air density’, ‘The mass density of the air in terms of weight per unit volume.’, 7, 4),

Often the data has the form:

advection- The horizontal transport of air or atmospheric properties. In meteorology, sometimes referred to as the horizontal component of convection.
advection fog- Fog resulting from the transport of warm, humid air over a cold surface.
air density- The mass density of the air in terms of weight per unit volume.

So replacing the hyphen and space with ‘, ‘ separates the term from the definition for the database. BBBEdit and TextWrangler let you find lines containing any set of characters so you can easily find all of the lines that didn’t get converted. Maybe there was a space after the hyphen. Or maybe the hyphen didn’t get copied.

Sometimes words get hyphenated and the raw text looks like this:

altimeter setting- The value to which the scale of a pres- sure altimeter is set so as to read true altitude at field elevation.

When you do your substitution you end up with two sets of delimiters. They don’t easily let you search for lines that have one or more occurrences of a set of characters. However, there is an easy workaround. Do a find all for ‘, ‘. A new text window will appear that lists all of the occurrences of the search term. Copy that list to a new document. Process duplicate lines to a new document. The new document has all of the lines that contain more then one occurrence or your search term. Look them up and fix them manually.

Capitalization

I usually want the first word of the definition to be a capital letter. Turn Case Sensitive on and search for

, ‘[a-z]

replace it with

\U&

What grep does is look for all definitions that start with a thru z and because the a-z is in brackets you can replace what is found. The \U& says to take what you found in the brackets and upper case it.

I’ll want to add a new field to a table and I have the data in a flat file that I’ve been working on.
For example, we recently broke words into sounds and needed to add them back to the database.
The words look like this in the BBEdit file:

a • l o t
‘a • c re
ai m s

The easies way that I’ve found to bring the sounds into the larger table is to first create a new table with id and sounds as the fields, import the sounds data, and then merge the two tables. Note: This is how it works with PHP MyAdmin. It may be slightly different from the command line.

This code imports the data into the Sounds table. Note that it ends with a semi-colon and no comma.

INSERT INTO `Sounds` (`id`, `sounds`) VALUES
(1, 'a • l o t'),
(2, 'a • c re'),
(3, 'a f • t er'),
(4, 'ai m s'),
...
(478, 'c o m • p o s t');

Now I can merge the sounds into another table in the same database by using the UPDATE command. You must include both tables after UPDATE, even though you are only updating data in one of the tables. Note that when using PHP MyAdmin you need to indicate the fields with the table name in backquotes, a dot, and then the field in backquotes. This may be different in the command line.

UPDATE `Pictures`, `Sounds` 
SET `Pictures`.`sounds`=`Sounds`.`sounds` 
WHERE `Pictures`.`id` = `Sounds`.`id`;

Here’s another example.

UPDATE `Pictures`, `ArticIV_Phonemes`  
SET `Pictures`.`concatenated_phonemes` = `ArticIV_Phonemes`.`Concatenated_phonemes` 
WHERE `Pictures`.`id` = `ArticIV_Phonemes`.`id`