The GoDaddy DNS event (which I wrote about here) has been the subject of many a post-mortem and water-cooler conversation in the web performance world for the last week. In addition to the many well-publicized issues that have been discussed, there was one more, hidden effect that most folks may not have noticed – unless you use Firefox.
Firefox uses OCSP lookups to validate the certificate of SSL certificates. If you go to a new site and connect using SSL, Firefox has a process to check the validity of SSL cert. The results are of the lookup cached and stored for some time (I have heard 3 days, this could be incorrect) before checking again.
Before the security wonks in the audience get upset, realize I’m not an OCSP or SSL expert, and would love some comments and feedback that help the rest of us understand exactly how this works. What I do know is that anyone who came to a site the relied on an SSL cert provided and/or signed by GoDaddy at some point in its cert validation path discovered a nasty side-effect of this really great idea when the GoDaddy DNS outage occurred: If you can’t reach the cert signer, the performance of your site will be significantly delayed.
Remember this: It was GoDaddy this time; next time, it could be your cert signing authority.
How did this happen? Performing an OCSP lookup requires a opening a new TCP connection so that an HTTP request can be made to the OCSP provider. A new TCP connection requires a DNS lookup. If you can’t perform a successful DNS lookup to find the IP address of the OCSP host…well, I think you can guess the rest.
Unlike other third-party outages, these are not ones that can be shrugged off. These are ones that will affect page rendering by blocking the downloading the mobile or web application content you present to customers.
I am not someone who can comment on the effectiveness of OCSP lookups in increasing web and mobile security. OCSP lookup for Firefox are simply one more indication of how complex the design and management of modern online applications is.
Learning from the near-disaster state and preventing it from happening again is more important that a disaster post-mortem. The signs of potential complexity collapse exist throughout your applications, if you take the time to look. And while something like OCSP may like like a minor inconvenience, when it affects a discernible portion of your Firefox users, it becomes a very large mouse scaring a very jumpy elephant.
Context is everything. Where you stand when reading or watching something shapes the way you experience it. Just as Einstein explained to us in the Train/Platform Thought Experiment, the position of the observer dictates how the event is described and recorded.
There is no difference with web performance. When a company develops an online application and presents it to customers (it doesn’t matter if they are outside/retail or inside/partner/employee), the perspective of the team that approved, created, tested, and released the application becomes, as a VP at a previous company explained to me, “interesting, but irrelevant”.
Step away from the world of online application performance for a minute, and put yourself in the shoes of the customer; become a consumer. How do you feel when a site, application, or mobile app is slow to give you what you want? I’ll give you some idea:
The stress levels of volunteers who took part in the study rose significantly when they were confronted with a poor online shopping experience, proving the existence of ‘Web Stress’. Brain wave analysis from the experiment revealed that participants had to concentrate up to 50% more when using badly performing websites, while EOG technology* and behavioural analysis of the subjects also revealed greater agitation and stress in these periods. (“Web Stress: A Wake Up Call for European Business”, emphasis mine)
I know it comes from a competitor, but it is true. It applies to me; it applies to you. And web performance professionals need to step away from the screens for a minute and put themselves in the shoes of the people standing on the platform.
Everyday, your online applications change, grow, fail, falter, and evolve – the train is always moving. To the people on the platform, all they see is your train and how it’s moving compared to the other trains they have watched go by. You worked hard on your train, polishing the brass, adding new cars, even upgrading the engine. To you, the train is a magnificent achievement that everyone should admire, especially now that the new engine makes it so much faster!
The customer on the platform is measuring how your updated train is moving compared to the MAGLEV bullet train on the super-conducting rail next to you and asking “How come this train is so slow?”
The complexity of a modern web site is astounding, and improving performance by 0.4 seconds is often a feat worthy of applause…among web performance professionals. From the perspective of your customers, that 0.4 second improvement is still not enough.
Web performance is a numbers game. As an industry, we have been focused on one set of numbers for too long. The customer experience, not the stopwatch, has to drive your company to the next level of performance maturity. To do that, you have to step off your online application train and take a cold hard look at what you deliver to your customers, alongside them down on the platform.
Had a great conversation with a colleague today. She and I were bouncing around some ideas, and I listed my top 3 topics in Web performance as “Speed, Revenue, and Experience”. She was quick to correct me.
“No, not revenue, conversions”.
She was right. Just last week, I talked about how critical it is to convert visitors into customers. Doing this in some businesses doesn’t mean that there is any revenue, but the goal remains the same.
Speed is the one everything thinks is the same as Web Performance. It’s not. It’s the don’t be that guy measure of Web Performance, the one that can be easily quantified and put on display. But performance for an online application is so much more than raw speed.
Experience is the hardest of the three to measure, because what it is depends on who you ask. Is it design, flow, ease of use, clarity, or none of these things? But a fast application can still make people cranky. There are online applications that are clearly designed to make the customer do things the way the vendor demands and these are the ones that make you go “Why am I here?”.
Now, can all the metrics that measure Web Performance be distilled to Speed, Conversions, and Experience? If you stepped away from the very product specific terms the Web Performance industry uses every day, what would describe the final, bottled, and served essence of Web Performance?
In February 2010, Fred Wilson spoke to the Future of Web Apps Conference. He delivered a speech emphasizing 10 things that make a Web application successful.
The one that seems to have stuck in everyone’s mind is the first of these. People have focused, quoted, and written almost exclusively about number one:
Fred has worked with Web and mobile companies for many years, so he comes at this with a modicum of experience. And for years, I would have agreed with this. But Fred goes on to describe 9 other items that don’t get the same Google-juice that this one quote does. There are probably 10 more that companies could come up with.
But a maniacal focus on speed means that in some companies, all else is tossed in order for that goal of achieving some insane, straight-line, one-dimensional goal. Some companies are likely investigating faster than light technologies to make the delivery of online applications even faster.
Can you base your entire business on having the fastest online application? What do you have to do to be fast?
Strip it down. Lose the weight, the bloat, the features. And what’s left is a powerful beast designed to do one party trick, likely at the expense of some other aspect of the business that supports the application.
If a company focuses on a few metrics, a few key indicators, they might evolve up to NASCAR, where it is not just speed, but cornering, that matters. Only left-hand corners, mind you, but corners nonetheless. Here speed is important, but is balanced against availability and consistency to ensure that a complete view of the value of the site is understood.
But is that enough? Do your customers always want to go left in your application? What happens if you are asked to allow some customers to go right? Do all of the other performance factors that you have worked on suddenly collapse?
As you can tell, growing up means that my taste in fast cars and racing forms has evolved, become more complex. Straight-line speed, followed by multi-dimensional perspectives have led me to realize that speed is only one feature.
So, if top-fuel and stock-car racing aren’t my gig, what is?
For a number of years in the 1980s and again since 2008, I have had a love of Formula One. The complexity of what these machines are trying to achieve boggles the mind.
Formula One is speed, of that there is no doubt. But there is cornering (left and right), weight distribution, brake temperature, fuel mix, traffic, uphill (and downhill, sometimes with corners!), street courses and track courses. And there are 24 answers to the same question in every race.
And then, there is a driver. In Formula One, a driver with an “inferior” car can win the day, if that inferiority is what is particularly suited to that course, in the hands of a skilled manager.
There is no doubt that like Formula One, speed is key to coming out on top. But if the organization is focused solely on speed, then your view of performance will never evolve. The key to ensuring a complete Web performance experience is a maniacal focus on a matrix of items: speed, complexity, third-parties, availability, server uptime, network reliability, design, product, supply-chain, inventory management integration, authentication, security, and on and on.
The Web application is a just that: a web. Multi-dependent factor and performance indicators that must be weighed, balanced, and prioritized to succeed. No web application, no online application, fixed or mobile, will survive without speed.
However, if speed is all you have, is that enough to keep someone coming back?
Is your organization saying that speed is all there is to performance?
SJE is an excellent supporter of the online economy. However, she is also very focused on the experience she suffers through on many online retail applications. The question I get frequently from the other end of the living room (Retail and Wardrobe Management Control Center – see image) is: “Is Company X a customer? Because their site (is slow | is badly designed | doesn’t work | sucks)!”.
Most of the time, there isn’t much to do, and the site usually responds and SJE is able to complete the task she is focused on.
Last night, however, a retailer did something that strayed into new territory. This company unwittingly affected the customer experience to such a degree that they actually destroyed the trust of a long-term customer.
This isn’t good for me, as I wear a lot of fine products from this retailer. But even in my eyes, they committed a grievous sin.
This retailer decided, for reasons that are known only to them, to delete a number of negative comments, reviews, and ratings for a product that they have for sale.
I just checked, and sure enough, all of the comments, including my wife’s very strong negative feedback about the quality, are gone.
I can think of a number of really devious and greedy reasons why a company might do this. It could also be an accident. If it was an accident, you might want to note that reviews and comments for this product were accidentally lost.
Now, if you went to a retailer and saw that your comments and reviews had been deleted, how would you feel? Would you trust that retailer ever again? What would happen if the twittering masses picked up the meme and started to add fuel to the bonfire?
A strong business, a solid design, an amazing presentation, and unrivaled delivery aren’t enough for some businesses. As a company, there is substantial effort, time, and treasure dedicated to converting visitors into customers. And it sometimes takes only one boneheaded move to turn a customer into the anti-customer.
This Post Rating:
- The Business. The reason that resources and people have been gathered together. There is a vision of what the group wants to do and what success looks like.
- The Design. Don’t think style; think Design/Build. This is where the group takes the business idea and determines how they will make it happen, where the stores will be, what a datacenter looks like, who they will partner with.
- The Presentation. How the Business and the Design are shown to people. How the shelves are stocked, the landing pages look, the advertising is placed, how the business looks to potential customers.
- The Delivery. This is the critical part of how the business uses the systems they have designed and the presentation they have crafted to deliver something of value to the potential customer.
Without any one of these, an organization will fail to meet the most critical goal it has set to be successful: an experience that turns a visitor or browser into a customer.
All the Business and MBA grads in the audience are yawning, and slapping their Venti non-fat, no-whip, decaf soy lattés down on the table. This message isn’t for you. Well, it is, but you can stand up and give your chair to one of the people behind you.
Now that I have Dev, QA, and Operations sitting with me (remember, the Business guys are still in the back of the room, tapping away on their Blackberries), tell me what you think of this conceptual table. How does the Table of Customer Experience relate to you?
Ok, put down the Red Bulls and Monsters and listen: Everything that Dev, QA, or Operations does has an effect on the experience (negative or positive) of the potential customer. If one of the table legs is broken (or even shorter than the others), the rippling shockwaves will eventually affect the entire operation.
So, if I were to ask the member so of your organization how their daily activities supported the online application in each of these four areas, do you think they could answer?
Grab a white board. This is going to be a long day.
Picture courtesy of sashafatcat
Someone walks into your store. They say hello, poke around the racks, ask a few questions. Then they walk out.
Now, if I asked you, how would you describe that person?
Customer? Visitor? Yes?
I have been asking this question in preparation for some session for a group of motivated partners and employees in Singapore and Bangalore. As I prepare the presenter slides (not the dense textbook slides the participants get – thank you Garr Reynolds!), I keep correcting the words, typing customer to describe a visitor who is not.
When you and your teams discuss deep topics like conversion rates and transaction abandonment (WAKE UP! NO MEDITATION!) does the group classify non-buying, real people as customers or visitors?
The label customer should be reserved for those visitors who complete the transaction and provide the revenue/information to the company whose online application they are interacting with. This means that the customer is the visitor who has bought into the entire online application experience.
A visitor becomes a customer only when they are happy with:
- The Business
- The Design
- The Presentation
- The Delivery
Where in the four areas has your application let the company down before?
If you asked a random visitor why they haven’t become a customer, what do you think the typical answer would be right now? Next week? A year from now?
Then ask your parents (or your spouse, if you’re brave) to use your application. You must show incredible restraint during this exercise (I suggest a remote operated camera and 6,000 miles of separation) to stop yourself from leaping in and telling them what to do, shaping their experience and guiding them to your expected and desired outcome.
Can they do it? Would your parents or spouse become a customer?
When you look at your online applications tomorrow, use beginners mind to truly look at what you are doing in the four key areas. If you find yourself shaking your head and saying that this doesn’t make sense, put yourself in the visitors’ shoes.
You may ask yourself if the application you provide to support your business is truly improving the visitor experience. What you have a strong chance of finding is that your application is designed for customers at the expense of visitors.
When a visitor doesn’t complete the tasks you defined for them to reach the goal of becoming one of your customers, what do you call them?
And do you know what to do next?
When I asked if traditional Web performance still mattered, the post generated a flurry of comments and questions that I haven’t seen in in a long time.
After some reflection and discussions with people who have been tackling this problem for longer than I have, the answer is yes, it does matter. However, synthetic Web performance measurement will not matter the way it does now. The synthetic approach will decrease in importance within fully evolved companies, organizations that have strong cultures of Web performance.
In these organizations, the questions change as the approach becomes foundational and integral to the operation of the online business. Ways of examining competition and performance improvement evolve, and the focus moves – from the perspective of We have a problem to one of of Our customers / visitors have a problem.
The shift is fundamental and critical. For as long as I have been in the business, synthetic measurements have served as a proxy for customer experience. But unless you get into the browser, out to where and how the customer uses the online application, the margin of error will remain large.
The customer is not an operational issue. There is no technical fix for perceived performance.
There is no easy solution for evolving the experience of performance.
Image courtesy of james_gordon_los_angeles
More than two years ago, I created a post that was frank in its statement that Web performance measurement isn’t just a technology issue, it’s a business issue.
As we approach 2012, a new question is driving how I examine the world I work in: Does traditional Web performance still matter?
Seems drastic and will raise the ire of more than a few folks I know, but it is a valid point of discussion. The entire Web performance industry needs to look around and determine how they got where they are and what the world will look like in 5 years.
The “Web” as it was defined when I started in the industry was simple – browser and page driven, with a growing focus on delivering services to visitors. Now, there is no definition of “Web” that can encompass everything that can be used when talking to companies. And in many cases, if asked, companies may not fully understand how customers interact with their online properties on a daily basis.
I used to be able to say what determined fast Web performance. Now, the simple answer is irrelevant, replaced with the reality of “It depends”. Fast is completely dependent on what is being done, when and where is it happening, how things being done, and who is driving the way it is is done.
I am issuing a challenge to the entire Web performance industry: Step back and and ask yourself if we are asking and answering the right questions for the companies we work with.
If we don’t find out now, in 5 years it won’t matter.
People in every organization are happy (in an unhappy way) to tell you exactly what their level of Web performance pain is. They go into great detail on how every performance issue affects them and and why it makes every day an unpredictable and almost unmanageable challenge.
If you take the personal perspective of Web performance pain, the risk not finding the real problem, the true cause of the pain.
Talking to customers at all levels of organizations has shown that when you ask “where it hurts”, they can tell you exactly what they want you to work on. And once you solve that problem, you get another person from the same organization with a different pain coming to you, complaining that you have ignored them.
A whole-organization focus is required when working to solve a customers Web performance pain. And it starts by asking questions of everyone in a company, not just the one who came to you for the initial diagnosis. Different groups at different levels have different questions.
Here’s a (very basic) list of some of those that you should be prepared to answer as you work to diagnose a company’s Web performance issues.
- How am I doing against my competitors?
- How does performance affect my revenue?
- If I want to use the Web for more revenue, what do I need to do to make it work?
- How does Mobile deliver what I need?
- How much will it cost me to deliver the necessary Web performance?
- What is critical for me to deliver now, and what can I delay until the next budget cycle?
- How do I ensure that Web performance issues don’t affect revenue?
- Are my partners helping or hindering us?
- How do I get Marketing to the table to understand the technology boundaries we have?
- How do I effectively use the Web without alienating customers with slow performance?
- How do I ensure that our design is delivered appropriately to both fixed-Web and mobile users?
- What parts of the site are customers unsatisfied with due to performance?
- Do my promotions scale to handle the surge in customers?
- How do I get Operations to understand that delivering new experiences with leading-edge technology is critical for us to be successful?
- I spend most of my time on troubleshooting conference calls. How can I reduce this drain on my time and resources?
- My team spends most of its time trying to correlate data between 5 different systems. Help!
- The latest design is putting a massive strain on our infrastructure. Didn’t anyone test this on the production servers before it went live?
- I know that we need to take a load of our servers, but I don’t know how to choose a CDN. What do I need to do?
- Man, I get a lot of alerts. How do I tell which ones I need to care about?
- This sure looks like a problem. How do I show the appropriate folks that this issue is their responsibility?
- Most of the time, the issues I investigate are with one third-party. Who is responsible for fixing this and does it really affect customers?
- I get bonused on fast MTTR. How can I figure out what the problem is faster?
In the sections above, notice that none of the questions need to be answered with product descriptions. Companies are desperate to understand not how other companies deployed the latest Kazoo to solve their Waka-waka problem, but how they made life easier and more manageable.
Coming to the customer with an open mind and a listening ear is the new hallmark of Web performance.