Performance Trends for ???? – Smarter Systems

IF-repair by Yo Mostro (Flickr)Most of the trending items that I have discussed in the last two weeks are things that can be done today, problems that we are aware of and know need to be resolved. One item on my trend list, the appearance of smarter performance measurement systems, is something the WPO industry may have to wait a few years to appear.
A smarter performance measurement system is one that can learn what, when, and from where items need to be monitored by analyzing the behavior of your customers/employees and your systems. A hypothetical scenario of a smarter performance measurement system at work would be in the connection between RUM and synthetic monitoring. All of the professionals in WPO claim that these must be used together, but the actual configuration relies on humans to deliver the advantages that come from these systems. If RUM/analytics know where your customers are, what they do, and when they do it, then why can’t these same systems deploy (maybe even create and deploy!) synthetic tests to those regions automatically to capture detailed diagnostic data?
Why do measurement systems rely on us to manually configure the defaults for measurements? Why can’t we take a survey when we start with a system (and then every month or so after that) that helps the system determine the what/when/where/why/how of data and information we are looking to collect and have the system create a set of test deployment defaults and information displays that match our requirements?
The list of questions goes on, but they don’t have to. Measurement systems have, for too long, been built to rely on expert humans to configure and interpret results. Now we have a chance to step back and ask “If we built a performance measurement system for the a non-expert, what would it look like?”
More data isn’t the goal of performance measurement systems – more information is what we want.

Managing Performance Measurement: Who uses this stuff anyway?

Clogged Pipe - staale.skaland - FLICKROne of the least glamorous parts of managing performance measurement data is the time I have to take every month to wade through my measurements and decide which stay on and which get shut off. Since I’m the only person who uses my measurement account, this process usually takes less than 10 minutes, but can take longer if I’ve ignored it for too long.
With large organizations that are collecting data on multiple platforms, this process may be more involved. By the time you look at the account, the tests have likely accumulated for months and years, collecting data that no one looks at or cares about. They remain active only because no one owns the test and can ask to disable it.
What can you do to prevent this? Adding some measurement management tasks to your calendar will help prevent Performance Cruft from clogging your information pipes.

  1. Define who can create measurements. When you examine account permissions on your measurement systems, do you find that far more people than are necessary (YMMV on this number) have measurement creation privileges? If so, why? If someone should not have the ability to create new measurements, then take the permissions away. Defining a measurement change policy that spells out how measurements get added will help you reduce the amount of cruft in your measurement system.
  2. Create no measurement without an owner. This one is relatively easy – no new measurement gets added to or maintained on any measurement system without having one or more names attached to it. Making people take responsibility for the data being collected helps you with future validations and, if your system is set up this way, with assigning measurement cost to specific team budgets. It’s likely that management will make this doubly enforceable by assigning the cost of any measurement that has no owner to the performance team.
  3. Set measurement expiry dates. If a measurement will be absolutely critical during  only a specific time range, then only run the measurement for that time. There is no sense collecting data for any longer than is necessary as you have likely either stored or saved the data you need from that time for future analysis or comparisons.
  4. Validate measurement usage monthly or quarterly. Once names have been associated to measurements, the next step is to meet with all of the stakeholders monthly or quarterly to ensure that the measurements are still meaningful to their owners. Without a program of continuous follow-through, it will take little time for the system to get clogged again.
  5. Cull aggressively. If a measurement has no owner or is no longer meaningful to its owners, disable it immediately. Keep the data, but stop the collection. If it has no value to the organization, no one will miss it. If stopping the data leads to much screaming and yelling, assign the measurement to those people and reactivate.

Managing data collection is not the sexiest part of the web performance world, but admitting you have a data collection cruft problem is the first step along the path of effective measurement management.