Web Performance Concepts Series – Revisited

Two years ago I created a series of five blog articles, aimed at both business and technical readers, with the goal of explaining the basic statistical concepts and methods I use when analyzing Web performance data in my role as a Web performance consultant.

Most of these ideas were core to my thinking when I developed GrabPERF in 2005-2006, as I determined that it was vital that people not only receive Web performance measurement data for their site, but they receive it in a way that allows them to inform and shape the business and technical decisions they make on a daily basis.

While I come from a strong technical background, it is critical to be able to present the data that I work with in a manner that can be useful to all components of an organization, from the IT and technology leaders who shape the infrastructure and design of a site, to the marketing and business leaders who set out the goals for the organization and interact with customers, vendors and investors.

Providing data that helps negotiate the almost religious dichotomy that divides most organizations is crucial to providing a comprehensive Web performance solution to any organization.

These articles form the core of an ongoing series of discussion focused on the the pitfalls of Web performance analysis, and how to learn and avoid the errors others have already discovered.

The series went over like a lead balloon and this left me puzzled. While the basic information in the articles was technical and focused on the role that simple statistics play in affecting Web performance technology and business decisions inside an organization, they formed the core of what I saw as an ongoing discussion that organizations need to have to ensure that an organization moves in a single direction, with a single purpose.

I have decided reintroduce this series, dredging it from the forgotten archives of this blog, to remind business and IT teams of the importance of the Web performance data they use every day. It also serves as a guide to interpreting the numbers that arise from all the measurement methodologies that companies use, a map to extract the most critical information in the raging sea of data.

The five articles are:

  1. Web Performance, Part I: Fundamentals
  2. Web Performance, Part II: What are you calling ‘average’?
  3. Web Performance, Part III: Moving Beyond Average
  4. Web Performance, Part IV: Finding The Frequency
  5. Web Performance, Part V: Baseline Your Data

I look forward to your comments and questions on these topics.

The Dichotomy of the Web: Andy King’s Website Optimization

Andy King's Website Optimization, O'Reilly 2008The Web is a many-splendored thing, with a very split personality. One side is drive to find ways to make the most money possible, while the other is driven to implement cool technology in an effective and efficient manner (most of the time).

Andy King, in Website Optimization (O’Reilly), tries to address these two competing forces in a way that both can understand. This is important because, as we all know from our own lives, most of the time these two competing parts of the same whole are right; they just don’t understand the other side.

I have seen this trend repeated throughout my nine years in the Web performance industry, five years as a consultant. Companies torn asunder, viewing the Business v. Technology interaction as a Cold War, one that occasionally flares up in odd places which serve as proxies between the two.

Website Optimization appears at first glance to be torn asunder by this conflict. With half devoted to optimizing the site for business and the other to performance and design optimization, there will be a cry from the competing factions that half of this book is a useless waste of time.

These are the organizations and individuals who will always be fighting to succeed in this industry. These are the people and companies who don’t understand that success in both areas is critical to succeeding in a highly competitive Web world.

The first half of the book is dedicated to the optimization of a Web site, any Web site, to serve a well-defined business purpose. Discussing terms such as SEO, PPC, and CRO can curdle the blood of any hardcore techie, but they are what drive the design and business purpose of a Web site. Without a way to get people to a site, and use the information on the site to do business or complete the tasks that they need to, there is no need to have a technological infrastructure to support it.

Conversely, a business with lofty goals and a strategy that will change the marketplace will not get a chance to succeed if the site is slow, the pages are large, and design makes cat barf look good. Concepts such HTTP compression, file concatenation, caching, and JS/CSS placement drive this side of the personality, as well as a number of application and networking considerations that are just too far down the rat hole to even consider in a book with as broad a scope as this one.

Although on the surface, the concepts discussed in this book will see many people put it down as it isn’t business or techie enough, those who do buy the book will show that they have a grasp of the wider perspective, the one that drives all successful sites to stand tall in a sea of similarity.

See the Website Optimization book companion site for more information, chapter summaries and two sample chapters.

In praise of found technology and the waste we treat it

The title is a deliberate misspelling. An event in the last two weeks has got me thinking about EWaste, and the way it is treated in the US, and likely the entire developed world.

About two weeks ago, #1 son told my wife to “STOP THE CAR!” as they were driving down the road near our house. Thinking he was mad, she did. #1 son leaped out and returned to the car with an HP Pavilion Desktop, in the vain hope that it could replace his current dinosaur computer.

This morning, I completed the configuration process by adding a wireless network card to this machine and they are now up and running with a computer from 2006-07, rather than the one they had been on up until then, which was from 1999 (seriously).

This leaves me to wonder why someone would dispose of a machine that is still perfectly functional. A machine that could have been donated to anyone of a number of causes to help those far less fortunate than we are.

I may complain incessantly about my lack of a MacBook ($|Pro|Air), but in the area of technology, I am well off. I have an excellent pair of servers that host my sites. Have a number of older machines in my basement to serve a variety of purposes, including development. I have my personal laptop and a very powerful work-provided laptop. And my wife has the most powerful machine in the house, to get e-mail and cruise Craigslist.

We are not computer-challenged. Yet, I do not take the disposal of any of this technology lightly. If I do dispose of technology, it goes into the city garbage ONLY on hazardous waste days. If I can, I give the machines to organizations who can use even a very old machine.

The processing of EWaste is a shameful burden that the wealthy of the world impose and throw down to the down-trodden. We pass along the poisons to those who are least able to say no, without a second thought.

To find and reincarnate a computer on the street is the act of a truly geeky family. To have thrown the computer to the curb in the first place is a sign of the shameful ignorance in our society for what is done with EWaste.

Are you being a responsible computer owner, as a person or a corporation?

Streaming v. Torrent – The true promise of on-demand

Steve Gillmor comments on TechCrunch today that Comcast’s caps on bandwidth may finally drive people from the land of torrents, and to the land of streaming. [here]

While I agree that the promise of streaming is long overdue, there is the one area that streaming still can’t fill: The mobile viewer. I don’t mean folks on mobile phones, although with the growth of 3G in the Americas (Europe and Asia laugh at us in the area), the mobile market will become more important.

No, I am referring to the mobile, laptop-using traveller, mainly the business traveller, although leisure travellers are starting to take their laptops with them more often. Streaming doesn’t work on the road, in a hotel with a crappy connection, in an airport, or somewhere were the 3G isn’t 3G enough.

While streaming will become more prevalent, it won’t unseat the culture of Torrents for a few years yet. It will happen. But affordable, reliable connectivity saturation across the Americas has to occur first. And, in some ways, Comcast and the other providers are the ones hampering this process.

The focus of the connectivity providers on their dinosaur cage-match with the FCC has left them ignorant of the asteroid screaming toward them. In order to create a streaming market that they can profit from, they have to open the pipes, lower the costs, and increase the options for the consumer of their Internet/bandwidth services. If the connectivity they provide to the consumers can’t support the desire for the streaming economy, the Torrent reality will not fade away.

Streaming and the Beijing Olympics – Is it live? Or is it NBC?

Over at NewTeeVee, there is a detailed look at the way that the streams were served and who served them. They also have a great discussion of the P2P and Torrent downloads, serving as an alternate method of getting the high quality streams out to a larger audience.

But John Furrier poses an interesting question, just before he quotes the entire NewTeeVee post: What’s the largest audience for a live stream?

Most of the streams that were viewed by people at NBC and other media providers were delayed. And while the concept of streaming is still valid, it doesn’t fulfill the promise of an “as it happens” delivery of streaming media.

So folks, what is the largest live video stream audience?

Bylined Article up at StreamingMedia.com

My bylined article Industry Perspectives: Best Practices for Flawless Web Multimedia Streams is up! Go check it out!

The discussion centers around how to approach monitoring and measuring the performance of Streaming Media, an area that is far more challenging than traditional Web page and site performance measurement.

There are a number of challenges an organization faces when deciding to adopt some type of streaming media strategy. The main one is “Do we go it alone?”.

The article addresses a number of these areas.

Immigration: Thoughts on the Green Card Process – A Personal Story

green_card_mazeBeing a Canadian in the United States for the last nine years has been an interesting adventure, to say the least. Although our nations are neighbours, there can be two more different approaches to the same problem so close to each other.

I can’t claim to be an expert on how Canada processes new immigrants, but I can say that it has to be better than what has happened in the US.

Nine years ago, I started out on the NAFTA Free-Trade visa, the TN-1. This is a non-immigration visa, which is restrictive in the occupations that you can work in, but essentially allows highly-skilled Canadians, Americans, and Mexicans to work in the free trade zone, a term I use very loosely in this context.

In 2001, my status was migrated by my previous employer to an H1-B. This is a much more formal visa and is used for skilled workers from around the world. It is also infamous for its quota system, and is the bane of most (if not all) high-tech firms who insist on recruiting the best talent from around the world to work in the Unuted States.

In some respects, while the off-shoring trend that was so big a concern a few years back (still?) is founded on a number of different economic realities, the driving force was the restrictive nature of the H1-B visa. I encountered a version of this when I changed employers and transferred my H1-B from employer A to employer B. When I did this, I could not leave the US, for any reason, until I had my new H1-B without forfeiting the entire process.

One of the conditions I had for switching employers was that employer B would start the Green Card process for me and my family. This process alone has taken 3.5 years, and from what I can tell, being an employment-based application from a Canadian means that I haven’t had to wait nearly as long as some of the people who apply under other circumstances or from “less friendly” nations.

This process is approaching (we hope) its final phase, as there is talk from the people assisting us that there is a chance that we may be processed through the final stages in late 2008 or early 2009. But, as with all things related to this process, this is still very much speculative.

Whew.

So, as a citizen of the United States reading this, you are likely saying “So what?”, or “How does this affect me?”. Frankly, it doesn’t. But, in a fundamental way, it does.

As a nation built almost completely on immigration, the United States has become increasing isolationist, especially in its immigration policies. Mostly at a political level. Where the conflict appears to be developing is between the political agenda and the economic needs of the US economy. US firms are reliant on importing the best and the brightest from around the world. These same firms are now finding increasing resistance from these highly-skilled employees who are looking at the current state of the US economy and the incredibly restrictive immigration criteria, and choosing to walk away, or choose other more lucrative and less restrictive opportunities.

As a person involved in this process, I can say that up until late last year, when I recieved my EAD, I was in effect an indentured serf, beholden to the company for which I worked, which none of the options or flexibility that my US colleagues had available to them.

I own a house. My children go to school in the town where I own my house. I pay US, not Canadian, taxes. I pay property tax.

However, in the eyes of the United States government, I am considered “three-fifths of a person”. A person bound to this country but not of this country.

The truly American among you may say “Shut up and become a citizen”. I chose not to. I have chosen to retain my personal Canadian Identity, those things that I hold dear that separate Canadian and Americans. I retain my Canadian passport. My youngest son holds dual-citizenship.

I have chosen to make a life in the United States. However, the process that I have been involved in does not allow me to recommend this path to any other Canadians.

To other Canadians, I say: Stay home. Make Canada the best it can be. Make it a truly integrated player in the Global Economy.

To the United States, I say: Wake up. Your destiny has come, and gone. And the way you treat your immigrants is a clear demonstration of that.

Some have said that the United States is a fading empire, most-often compared to the Roman Empire. However, as this piece in the LA Times states, even an ancient empire, in its fading glory, understood how you become great: You become more than the sum of your parts.

WGBH – TV: Wasting airspace, again

WBGH, again you win the contest as the most clueless PBS station in the world.

There is this small news event occurring this week, called the Democratic National Convention.

And what are you showing tonight?

John Denver, a man and his music.

Are you completely clueless?

Another year, another lack of donations from me.

When will you start showing the programs we want to see, and stop with the crap you think raises you the most money?

This is an ongoing rant.

IP Addresses by Country and Registrar — August 2008

The five top-level IP Registrars (ARIN, RIPE, APNIC, LACNIC, AFRINIC) manage tens of million IPV4 and IPV6 addresses, assigning them to networks around the world. These addresses are key to accessing the Internet for all end-users.

About once a year, I generate the statistics out of the IP address database I maintain. The last one was published back in August 2007,  and, as can be imagined, there have been substantial changes that have occurred in the last 12 months.

At the country level, there has been a substantial change in the top five as China has superseded Japan in the total number of IPV4 addresses. In fact, in the last 12 months, Japan has seen a net loss in the total number of allocated IPV4 addresses.

country NETWORKS IPS
UNITED STATES 33758 1441234432
CHINA 1478 161991424
JAPAN 1971 148897280
EUROPEAN UNION 4766 120292412
UNITED KINGDOM 3460 85509464
GERMANY 2237 78525232
CANADA 5201 73586944
FRANCE 1692 68225984
KOREA, REPUBLIC OF 631 66108928
AUSTRALIA 5950 35859200
ITALY 921 29520320
BRAZIL 266 27657728
TAIWAN, PROVINCE OF CHINA 410 23842816
MEXICO 158 21504000
SPAIN 494 20755104
NETHERLANDS 1319 20614440
RUSSIAN FEDERATION 2654 19910472
SWEDEN 800 16947104
INDIA 542 16661504
SOUTH AFRICA 790 13828352
POLAND 1457 12799116
DENMARK 459 9149792
TURKEY 269 8892096
FINLAND 619 8812672
SWITZERLAND 1217 7642056
ROMANIA 693 7612416
HONG KONG 633 7537408
NORWAY 388 7119648
AUSTRIA 793 6938336
INDONESIA 429 6885376
ARGENTINA 380 6241280
BELGIUM 370 6016640
CZECH REPUBLIC 572 5592704
NEW ZEALAND 1187 5560064
VIETNAM 81 4388352
THAILAND 338 4367104
UKRAINE 1547 4356416
SINGAPORE 395 4352512
CHILE 380 4313344
IRELAND 204 4025760
MALAYSIA 238 3949312
ISRAEL 211 3936192
PORTUGAL 134 3859296
COLOMBIA 217 3818752
GREECE 220 3746816
VENEZUELA 109 3592704
HUNGARY 231 3577472
PHILIPPINES 257 2957056
BULGARIA 331 2923264
UNITED ARAB EMIRATES 23 2278656
EGYPT 85 2255616
SAUDI ARABIA 154 2194176
LITHUANIA 117 1997696
IRAN, ISLAMIC REPUBLIC OF 94 1588224
CROATIA 62 1438048
SLOVAKIA 157 1418240
COSTA RICA 24 1332224
LATVIA 187 1316864
PAKISTAN 110 1302016
PERU 74 1253120
SLOVENIA 187 1210752
PANAMA 51 1038592
ESTONIA 77 1008912

The adjustment in China is completely expected, as China has substantial room to expand its IPV4 population, compared to a country such as Japan which is likely approaching saturation in this area.

In the total number of IPV4 addresses assigned by registrar, ARIN (US, Canada, and parts of the Caribbean) still has a substantially larger population than the other regions. However, this region only grew by 3 million IPV4 addresses in the last 12 months, compared to 42 million for RIPE (Europe and the Middle East), and 74 million for APNIC (Asia-Pacific). LACNIC (Mexico, Latin and South America) grew by 13 million IPV4 addresses and AFRINIC (Africa) by 4 million.

registry NETWORKS IPS
arin 39231 1583464704
ripencc 29996 539874744
apnic 15143 480405504
lacnic 2152 74576896
afrinic 1325 20201216

Some of this growth in APNIC, AFRINIC, and LACNIC can be attributed to adjustments in the Registry structure itself. Now that there are five registrars with responsibility for a clearly defined set of nations, a number of adjustments have had to occur in the registrar of record for IPV4 ranges.

In the past, for nations that were not explicitly covered by a regional registrar, the IPV4 range may have been assigned by the registrar now responsible for the region, most likely ARIN or RIPE.

Now that there are registrars specifically designated to cover these formerly grey areas, these historical artifacts are being corrected as registry information is renewed.

However, this cannot directly account for the noticeable growth in AFRINIC, APNIC, and LACNIC relative to the two older registrars. This is yet another indication of the strong growth of the Internet outside of its established base of the United States, Canada, Western Europe, and a small number of industrialized nations in the Asia-Pacific region.