Watch for the surprises

Look for and act on the surprises around you ever day.  That’s where we have the most opportunity to make changes for the better.

Yesterday, amidst all the discussion of CERN’s announcement that it seemed to have measured neutrinos moving faster than light, Mark Jason Dominus reminded me of the line often attributed to Asimov that the sound of scientific breakthrough is not “Eureka!” but “That’s funny…”

Back in the late 90s, I had a new boss in the IT department. When he called his first staff meeting, he told us to bring some sort of metrics from our areas of responsibility.  I’d recently created the company’s intranet, so I ran some reports of hits by top-level directory from the Apache logs.  My boss gave my stats a cursory glance, handed them back and asked “So what surprised you?”  The question itself surprised me.  “Now that I look at it,” I answered, “I didn’t expect that the /foo directory would be getting so many hits.”  Then came his crucial follow-up: “So what should we do about it?”

So what’s going on around you that you didn’t expect?  Are you even looking?  How?  Where?

Look at your server log files.  Try to absorb patterns.  Who’s requesting the same non-existent .GIF file every 11 minutes?

Run a profiler on your code. Why is the sort function being called so many times?  Why would a simple string transformation function take so long to execute?

Measure your system performance with a tool like Munin or Cacti.  Look for use spikes.  What happens at 3:30am that thrashes the system?  Why do the cache hits drop to near zero twice a day?

Always keep an eye open for the unexpected behavior, the strange blip, the neutrino that took 60 nanoseconds longer to get there than expected.  Then follow up on what you find.

Advertisements

Tags: , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: