Thought Delimited

Last month, during a brief lull in my work projects, I decided to tackle an issue I'd been putting off for awhile:  finding a better way of managing errors caught and reported by my ColdFusion applications.  All of my apps are designed to email me a full report of any errors that reach the global error handler, and while that means I'm immediately notified if there's a problem I need to address, the error message folder in my email account ends up serving as a de-facto error archive.  And now that we're (slowly) moving towards building applications as a team, it makes sense to have the errors stored in a centralized location.

After doing a bit of research, I ended up at the home page for the open-source bug logger BugLogHQ, created by Oscar Arevalo.  Based on the project changelog list, it sounded like it might have the features I wanted, but there wasn't a lot of explanatory documentation on the site and the screenshot images were thumbnails that couldn't be enlarged.  And it didn't support Oracle, which is our primary database engine at work (and a perusal of the Google Groups forum for BugLogHQ confirmed that).

Despite those factors, I decided it was worth trying to add Oracle support to the codebase so I could run it and try it out.  I'm glad I did:  BugLogHQ is a well-designed, robust application for managing errors in a way that will still let me keep track of what's happening with my apps.  Now that I've seen how it operates and what it can do, I'm kind of surprised that I had to do research to come across it:  either it completely slipped under my radar or it needs more publicity/exposure in the community.

Several things I like about BugLogHQ:

• I like the fact that it's easy to get up and running.  Download the files, put them in a "buglog" directory under the webroot (or use a mapping), run the SQL install script for your database of choice (mySQL, PostgreSQL, MS SQL 2000/2005, MS Access, and now Oracle) update the config file with your datasource name and database engine, and simply point your web browser to that "buglog" directory.  After logging in (which forces you to change the default password for the built-in admin user account), you can verify everything's working by going into the settings area of the app and running the tests, which all mimic the methods your other applications (referred to in BugLog parlance as "client" apps) can use to transmit errors to BugLogHQ.  Simple and straightforward.

• Just like most email clients have rules for filtering and acting on incoming email messages, BugLogHQ provides some rules for managing the errors that come in.  You can set up multiple "mailAlert" rules that will forward on particular errors (filtered by error type/severity, the source app's hostname or application name, or keywords in the configured error message text) to an email address of your choosing, or you can create a mailRelay rule that relays every error received to an email address.  And the emails sent out by BugLogHQ aren't just notifications about the errors:  they provide the full error details so you don't have to log into BugLogHQ to get more information.  There are also rules for discarding certain errors or for monitoring "heartbeat" transmissions from an app, so if the app is offline for a certain length of time BugLogHQ can report the problem.

• The methodology BugLog uses to report errors is very well-designed.  If your client app is also a CFML app, you simply instantiate a single CFC (with settings for interacting with the BugLogHQ web service you want to utilize) in either the application scope or a bean factory, then call it with its notification function whenever you want to send information to the BugLogHQ app.  If the client app is reporting an error, you can send BugLogHQ a line of message text, the error struct (the raw "cfcatch" struct), the type/severity of the error, and a struct of any additional information you want as part of the report (like the affected user's ID or name). Here's how I called it from the main.error() controller function in the FW/1 application I used during my testing (where I've instantiated the CFC as a service object):

  public void function error(rc) {
      var extraInfo= {username= session.user.getUsername()};
      variables.bugLogService.notifyService("Error report from TestApp1",request.exception,extraInfo,"INFO");
  }
  

  All of those parameters are optional, so you don't need an error struct if you're reporting something that's a status or informational message rather than an error.  If there's a problem with sending the report to BugLogHQ, it'll fallback to sending an email containing all that information to whatever address you specify so the error doesn't get lost.

• I also appreciate how BugLogHQ "eats its own dogfood".  If an error occurs in BugLogHQ itself, it'll record the error just like it would for any client app, and if the error occurs during the recording process it'll report the error via email.

• BugLogHQ utilizes a scheduled task (with a default interval of 2 minutes) referred to in the app as the "BugLog Listener Service" to denote and process incoming error messages.  When the task fires, the error messages that have been received since the last run of the task are processed and then written to the error database.  If there's a problem with writing to the database (which I understandably encountered as I was tweaking the code to support Oracle), it'll maintain the errors in the queue until the next iteration (and you can view the contents of the queue within the BugLogHQ dashboard).  Sometimes you need to restart that service in order for certain changes to take effect. If you somehow forget to reactivate the service, it'll be restarted when it receives a new error report from a client app.

• If one or more of your clients apps are NOT CFML-based web applications, there are library files provided that let you submit errors from PHP or Python apps, or straight from Javascript.  I haven't tried out those files yet, but we do have a few PHP apps in our shop now that could make use of the PHP option.

• There are configuration options for tying BugLogHQ to a JIRA instance, providing error summaries via RSS or a digest email, and setting an API key to ensure that only your apps can submit data to BugLogHQ.

Bottom line:  if you don't yet have an error-logging application or other mechanism for storing and managing errors from your application(s), you should take a look at BugLogHQ.  Oscar and his contributors have done a great job with this application.

In a recent post, I mentioned that the feature that allows a user to tap on a phone number in their smartphone web browser and dial that number doesn't work if the phone number contains letters.  Well, some of the phone numbers in the database tables I'm working with contain letters (usually to spell out building abbreviations) so I needed to convert those letters into the corresponding digits on American phones (apparently it's not an international standard).

Before writing my own function to do this, I did some Googling and was a bit surprised to find that while folks have written routines in CFML to strip out dashes and such, no one had really written (and then published) a routine for this task.  So here's mine:

public string function convertPhoneLetters(required string oldNumber) {
  var newNumber= arguments.oldNumber;
  var regArray= [
    "[A-Ca-c]",
    "[D-Fd-f]",
    "[G-Ig-i]",
    "[J-Lj-l]",
    "[M-Om-o]",
    "[P-Sp-s]",
    "[T-Vt-v]",
    "[W-Zw-z]"
  ];

  var resultArray= [
    2,3,4,5,6,7,8,9
  ];
		
  for (var x=1; x <= ArrayLen(regArray); x++) {
    newNumber= ReReplace(newNumber,regArray[x],resultArray[x],"all");
  }
		
  return newNumber;
}

(Shrug) Nothing fancy, but it works. If you need to do additional adjustments to the phone number (like I did), you can just add them to this function or put them in separate functions (which is probably better).

Just got back home from cf.Objective().  During the plane trip home, I wrote down a few paragraphs about the sessions I attended that I'll share now before kicking back for the rest of the day:

• Marc Esher's session on threads and concurrency was excellent.  If you implement or plan to implement concurrent code execution in your CFML applications, you need to look at the CFConcurrent library he developed (available on RIAForge and GitHub).  The library utilizes the Java Concurrency Framework (JCF) and makes it possible to return results/data from with a thread (something you can't do with <cfthread>), schedule/defer threaded actions, and set up "threading within threading."

• Nathan Strutz provided so much information and so many examples of using LESS to manage and generate CSS that I missed a few things trying to take notes.  No worries, though: he put up everything from his presentation on GitHub, including a number of examples and demos that show you some best practices on how to add LESS compilation into your development and production workflow.  And not long after the session, Matt Gifford made his own contribution to the LESS toolset:  a CF Builder extension that lets you do the compiling within the IDE.

• Curt Gratz's session on mocking covered the reasons why you should mock data and objects when you unit testing, then showed attendees how MockBox makes it easy to mock function calls and their results on the fly.  As someone still finding his way regarding unit testing, I found his one walkthrough on how to make sure you're always testing a single function, and not a parent or dependent function, very helpful.  And even though MockBox is part of the ColdBox family of projects, it can be run as a standalone product, so you can make it part of your unit testing strategy regardless of what framework you use.

• Billy Cravens also did an excellent job with his session on authenticating users via social networks.  Adding Google, Facebook, and Twitter-based authentication to your site is really now just a matter of getting an API and private key for your site from those services and making use of the available.  His demo code, available on GitHub, shows examples for all three service providers within a simple application built with the FW/1 framework.

• I attended Denard Springle's multi-factor authentication workflow session mostly out of curiosity, as it's not something I can really implement with my particular user base.  But he showed us his technique of securing URL and form variables submissions, which involves hashing the variable names and encypting the variable values.  That's something I've never seen anyone else do.  Slides from his demo that illustrate that technique are available from his collection of presentation slide decks (look for the "Multi-Factor Authentication" one).

• A room for conducting ad-hoc discussions or coding sessions was provided, and I ended up hosting a half-hour session on CFML applications in higher education.  The seven or so of us described some of the applications we've worked on, and as I suspected there was quite a diversity of applications, mostly built in support of the administrative processes at our schools rather than the educational process itself.  But I did learn that a few of us use (or are in the process of implementing) the Jasig CAS single sign-on tool, and that there would be interest in having some kind of robust form/survey tool (possibly with plugins like one to handle online payments) that we could set up such that non-programmers could create what they needed from it.  The question was raised as to whether it would make sense to augment Mura CMS to create such a thing.  We didn't discuss or make any further plans beyond that, so whether the discussion will continue in some form (move further forward) online remains to be seen.

As my conclusion, I will say that the conference was excellent as always (hardly a unique opinion, I know).  I've been to cf.Objective() before (in 2008), but this was my first time attending at the current venue, which is the Nicolett Mall area of Minneapolis.  It's a very pedestrian-friendly area of the city, and with the near-perfect weather we had it was quite lively.  It gave us a lot of options for dining and socializing after the sessions, and being able to hang out with other members of the community is one of the great benefits of the conference.

Several months ago, I was doing some functional testing on a web application that was taking forever.  Each run of the test resulted in changes to the data in several tables, and in order to reset the test I had to go into the tables and undo the changes.  It occurred to me that it would be nice to have a tool or script that could preserve the current table data and then later write that version of the data back to the tables.

That marked the beginning of the tableSnapshots tool.

Read more...

Every ColdFusion developer who spends a lot of time working with queries knows that if you have to run a unique query for every iteration of a loop, you should (whenever possible) use a <cfquery> to acquire the entire set of records prior to the loop and then query against that recordset using a Query of Queries (QofQ):

<cfquery name="qryMain" datasource="myDatabase">
    select a.field1, a.field2, b.field7 ...
    from table1 a, table2 b
    where a.field1= b.field2
</cfquery>

<cfloop index="j" from="1" to="1000">
    ....
    <cfset calculatedVal= thisVal + thatVal />
    
    <cfquery name="qrySub" dbtype="query">
        select field2, field7 ...
        from qryMain
        where field1= <cfqueryparam value="#calculatedVal#" cfsqltype="cf_sql_numeric" />
    </cfquery>
    ...
</cfloop>

...That way, you're querying the recordset in memory 1,000 times instead of making 1,000 calls to the database.

Yesterday I was asked to look at an old report that wasn't performing well anymore now that the database tables being queried had grown much larger. While it was designed to use QofQ data within the loop, the QofQ in the loop was doing a join between two recordsets stored in memory:

<cfquery name="dbQry1" datasource="myDatabase">
    select a.field1, a.field2, a.field3, b.field4, b.field5
    from table1 a, table2 b
    where a.field1= b.field2
    ...
</cfquery>

<cfquery name="dbQry2" datasource="myDatabase">
    select c.field1, c.field6, d.field9
    from table3 c, table4 d
    where c.field1= d.field3
    ...
</cfquery>

<cfloop index="j" from="1" to="1000">
    ....
    <cfset calculatedVal= thisVal + thatVal />
    
    <cfquery name="qrySub" dbtype="query">
        select dbQry1.field2, dbQry1.field3, dbQry2.field6, dbQry2.field9
        from dbQry1, dbQry2
        where dbQry1.field1= dbQry2.field1
        and dbQry.field2 > <cfqueryparam value="#calculatedVal#" cfsqltype="cf_sql_numeric" />
        ...
    </cfquery>
    ...
</cfloop>

I discovered that if I created a third QofQ that took care of the join outside of the loop, and queried that new recordset within the loop, the per-iteration performance improved dramatically:

<cfquery name="dbQry1" datasource="myDatabase">
    select a.field1, a.field2, a.field3, b.field4, b.field5
    from table1 a, table2 b
    where a.field1= b.field2
    ...
</cfquery>

<cfquery name="dbQry2" datasource="myDatabase">
    select c.field1, c.field6, d.field9
    from table3 c, table4 d
    where c.field1= d.field3
    ...
</cfquery>

<cfquery name="qryMasterSub" dbtype="query">
    select dbQry1.field2, dbQry1.field3, dbQry2.field6, dbQry2.field9
    from dbQry1, dbQry2
    where dbQry1.field1= dbQry2.field1
</cfquery>


<cfloop index="j" from="1" to="1000">
    ....
    <cfset calculatedVal= thisVal + thatVal />
    
    <cfquery name="qrySub" dbtype="query">
        select field2, field3, field6, field9
        from qryMasterSub
        where field2 > <cfqueryparam value="#calculatedVal#" cfsqltype="cf_sql_numeric" />
        ...
    </cfquery>
    ...
</cfloop>

It makes sense when you think about it, but I had just never considered that join operations would affect QofQs so drastically.