I’ve been working on a largish website project and I’ve been falling steadily behind schedule. The reasons for the delays fall into two categories: data conversion and code conversion. Both turned out to be more complicated than I expected.

Data Conversion (Filemaker to MySQL)

Filemaker is a weird sort of database. Instead of tables you have “layouts” that show certain fields but all fields exist in the same bucket. The database is the table. In this case the Filemaker database had hundreds of fields in it that were spread across 16 layouts and all of it needed to be converted to MySQL. Unfortunately this was not an a => b scenario for several reasons:

- A lot of calculations and data manipulation are taking place in Filemaker
- The LDML code made extensive use of lists built into Filemaker
- A lot of data had to be translated
- Certain restrictions were placed on the MySQL databases (see below)

The data conversion is part of the larger conversion from Filemaker/LDML to MySQL/PHP. On the PHP side the client already has a database structure and code built out for several of the portions of the site being converted. In those cases some data has to be split across multiple tables, so data has to be converted to new formats, and various other tweaks had to be performed.

This is a longer term project (four months). Because the data conversion will have to be done several times, I chose to write a script to handle it. Again, because it will have to be run over and over again, it first starts by emptying the existing tables and resetting all of the auto_increment counters to 1. When the site has been fully converted and is ready to relaunch, we’ll be able to run the script again and at the push of a button grab the most recent data from the previous site. Whenever data turns out to need more “massaging” during the code translation process (i.e. something turns out to be wrong) I can make the changes and re-run the script without having to perform a lot of manual labor.

I’ll outline the conversion script another day — it’s about 800 lines of code. It took about a week to write and get right and since this project is ongoing it will almost certainly require more tweaks as the project moves forward. That’s a lot more than the one day I had estimated for the conversion.

Code Conversion

Once I had the data poured into 19 MySQL tables I could start setting up the website. First thing was converting the new layout into XHTML and CSS. No problemo. The second step was to setup the new navigation structure. A little bit more work and planning here but again, no problemo. The third step was to create pages for all of the static content. This was the easiest step in the process. The fourth step was, and is, where I get my hands dirty and start mucking around with the site code.

Now, it seemed easiest to start with the pre-existing PHP modules from the client and get those up and running. Unfortunately, this is where I ran headfirst into one of my own great shortcomings. I have a hard enough time reading my own well-formatted and well-documented code. In this case I’m reading someone else’s code with functions and classes that are spread across 25 different files. Everything is being generated or manipulated by something else. Coming into this from the outside it’s very hard to figure out what is going on. For me this has led to a lot of confusion, a lot of delays, and a lot of sleepless nights.

Finally, this weekend I decided that something had to give. I was trying to take the short route and fumble through things. Define a task, tackle it, and when it doesn’t work try to trace the code back through the various files to find out why. Fix, try again, run into another problem, trace it back through, etc.

For most people this would actually work. They can just learn from doing something hands on. For me that doesn’t work. I have to first read a manual, then get hands on and refer back to the manual as I’m going along. It’s not enough for me to know how something works, I have to understand why it works.

I determined that I needed to write a manual. In doing so I would develop a deep understanding of how everything in the various files worked.

Yesterday I spent the day printing out most of a ream (500 pages) of code. I got a binder and a bunch of tabbed inserts. I created tabs for each document, hole punched everything and assembled my book. Then I started the two day process of reading the book. For each file I created an outline of each class and function. I also outlined the structure for each include. This is my table of contents and is in the front of the book.

Now I have a much greater understanding of what is going on because I’ve read through everything once. I also now have a quick reference to any section of code without having to open five different files to track function calls. At a glance I can see what is going on and if I’m passing data to a function in the correct format or using a function in the correct way. This took two days and it could have saved me weeks of sleepless nights.

Planning and organization are incredibly important when working on projects. It may seem like wasted time but in the end it saves more time than it consumes.

You can leave a response, or trackback from your own site.