The Blog Herald reports on how TypePad suffered a catastrophic drive failure this week that lead to some lost content.
Steve Rubel was one of those affected, and he got a very appropriate response from the CEO of TypePad to explain the situation.
GrabPERF caught the outage at about 21:00 EDT on October 4, 2005. Not a long one, but obviously long enough.
Bob Wyman of PubSub has a great post on the value of online consistency to companies and services. [here]
This is a critical component that is too often overlooked. The need for speed is an addictive goal. But as I have said before, fast is not enough.
Go, Bob, Go!
The weekly GrabPERF Search Index Results are in. Sorry for the delay.
Week of August 29, 2005 – September 4, 2005
TEST RESULT SUCCESS ATTEMPTS -------------------- --------- ------- -------- PubSub - Search 0.4132426 99.95 5561 Google - Search 0.5546451 100.00 5570 MSN - Search 0.7807107 99.87 5572 Yahoo - Search 0.7996602 99.98 5571 eBay - Search 0.9371296 100.00 5571 Feedster - Search 1.1738754 99.96 5569 Newsgator - Search 1.2168921 99.96 5569 BlogLines - Search 1.2857559 99.71 5571 BestBuy.com - Search 1.4136253 99.98 5572 Blogdigger - Search 1.8896126 99.74 5462 BENCHMARK RESULTS 1.9096960 99.79 75419 Amazon - Search 1.9795655 99.84 3123 Technorati - Search 2.7727073 99.60 5566 IceRocket - Search 5.0256308 99.43 5571 Blogpulse - Search 6.5206247 98.98 5571
These results are based on data gathered from three remote measurement locations in North America. Each location takes a measurement approximately once every five minutes.
The measurements are for the base HTML document only. No images or referenced files are included.
I know you have other things on your plate right now, but you seem to have a DNS problem.
redcross.org. 86400 IN NS arcdns3.redcross.org. redcross.org. 86400 IN NS arcdns2.redcross.org. redcross.org. 86400 IN NS arcdns1.redcross.org. ;; Received 172 bytes from 204.74.112.1#53(TLD1.ULTRADNS.NET) in 46 ms redcross.org. 3600 IN A 162.6.217.197 redcross.org. 3600 IN NS arcdns2.redcross.org. redcross.org. 3600 IN NS arcdns1.redcross.org. ;; Received 122 bytes from 162.6.217.142#53(arcdns2.redcross.org) in 33 ms
The name server arcdns3.redcross.org (162.6.90.141) either doesn’t exist, or is broken. Unfortunately, the Top-Level DNS servers still have it on file.
The site is working great [here]. Except for the DNS issues.
Keep up the good work.
This was interesting this morning.
Wonder what’s up…
UPDATE: Outage lasted from 00:30 to 10:30 Eastern Time, July 14, 2005.
Technorati: Walmart
I have been seeing these bursts of traffic, mainly from spambot morons, that have suddenly been crushing my server. The main cause: excessive database connections.
This was quickly remedied today when I changed all of the mysql_connect statements to mysql_pconnect statements. This allows PHP to use an existing connection to the MySQL database to serve requests from the same Apache child process.
Now the truly geeky among you are going “DOH! Wadda ya mean you were opening a new connection for every request?”. Well, believe it or not, I will bet you dollars to doughnuts that your blog app doesn’t persist database connections. Not a big deal if your database is on the same machine, and you are using local named pipes to make requests. However, if that database is located on another machine, if you do a netstat, you will see a large number of connection on port 3306.
Persisting database connections is particularly important for large hosted services. A great deal of TCP overhead, and kernel space memory can be saved by simply not letting the Web server saturate the database with individual database connections for every page request.
Without persistent database connections, eventually the TCP queue will be full of database connections and no one will be able to connect to the server, or they will get a lovely “can’t connect to database error”.
Ok, started to notice a dramtic and sudden increase in traffic to my site yesterday. Turns out that all of these folks were headed to the same place at this host:
/index.php?disp=stats
So, when I checked this out, they were all indicating referrals from the usual illicit medication and adult sites.
<sigh> More trackback and comment spam.
Now, I know that this page exists in b2evolution, and it is a way for visitors to view my traffic stats. However, a link to this page does not exist in my main display page. The only link to my stats is to my StatCounter stats.
Enter mod_rewite.
A simple rule disposes with these morons.
RewriteCond %{QUERY_STRING} disp=stats
RewriteRule ^.*$ http://www.pierzchala.com:9080/ [R,L,NS]
Please do not attempt to load the redirected URL; you will get nothing. NADA! That port is set to be dropped by iptables, effectively hanging the client end as it attempts to make a TCP connection.
/sbin/iptables -A INPUT -p tcp -i eth0 -s 0/0 --dport 9080 -j DROP
I use iptables to handle a lot of these morons. As the only people who view this page are infected with some virus or spyware, then I feel no shame in tying up their systems.
In the world of Web performance, the agreed upon state of Nirvana is the development of an automated system that will isolate, identify, diagnose and resolve (or suggest a resolution) to an issue. However, the question for me is whether these systems are really useful.
Why do I say that? Because they solve the tactical issues. The day-to-day issues. But there is no solution for poor design, inadequate eqipment, overloaded systems, and other strategic decisions. Automated performance systems do not solve the underlying problem — delivering reliable and relevant information on Web performance metrics that matter to business customers.
Who consumes Web performance data? Technology teams.
Who needs holistic Web performance information? Business leaders.
Who does the Web performance industry currently serve?
In writing the last post, I was thinking about what factors go into making the Web performance of a site “excellent”. What defines in the minds of the sites users/customers/visitors/critics/competitors that the performance of a Web site is excellent?
These are usually judged by the standard factors:
But within the company itself, how is the performance of their Web site judged to be excellent?
Right now, most people use the external metrics mentioned above to determine excellence. However, it must be remembered that there are two other critical factors that need to be considered when managing a large IT infrastructure.
When this two critical internal factors are combined with the raw external data that can be collected, collated and analyzed, some other ideas come to the forefront as KPIs in Web Performance Excellence:
The creation of new Web performance excellence metrics is crucial if companies truly want to succeed in the e-business arena. Business management has to demand that IT management become more accountable to the entire business, using metrics that clearly display the true cost of doing business on the Web.
Service Level Agreements (SLAs) appear to finally be maturing in the realm of Web performance. Both of the Web performance companies that I have worked for have understood their importance, but convincing the market of the importance of these metrics has been a challenge up until recently.
In the bandwidth and networking industries, SLAs have been a key component of contracts for many years. However, as Doug Kaye outlined in his book Strategies for Web Hosting and Managed Services, SLAs can also be useless.
The key to determining a useful Web performance SLA rests on some clear business concepts: relevance and enforceability. Many papers have been written on how to calculate SLAs, but that leaves companies still staggering with the understanding that they need SLAs, but don’t understand them.
Relevance is a key SLA metric because an SLA defined by someone else may have no meaning to the types of metrics your business measures itself on. Whether the SLA is based on performance, availability or a weighted virtual metric designed specifically by the parties bound by the agreement, it has to mean something, and be meaningful.
The classic SLA is average performance of X seconds and availability of Y% over period Z. This is not particularly useful to businesses, as they have defines business metrics that they already use.
Take for example a stock trading company. in most cases, they are curious, but not concerned with their Web performance and availability between 17:00 and 08:00 Eastern Time. But when the markets are open, these metrics are critical to the business.
Now, try and take your stock-trading metric and overlay it at Amazon or eBay. Doesn’t fit. So, in a classic consultative fashion, SLAs have to be developed by asking what is useful to the client.
Asking and answering these questions makes the SLA definition process relevant to the Web performance objectives set by the organization.
The idea that an SLA with no teeth could exist is almost funny. But if you examine the majority of SLAs that are contracted between businesses in the Web performance space today, you will find that they are so vaguely defined and meaningless to the business objectives that actually enforcing the penalty clauses is next to impossible.
As real world experience shows, it is extremely difficult for most companies enforce SLAs. If the relevance objectives discussed above are hammered out so that the targets are clear and precise, then enforcement becomes a snap. The relevance objective often fails, because the SLA is imposed by one party on another; or an SLA is included in a contract as a feature, but when something goes wrong, escape path is clear for the “violating” party.
If an organization would like to try and develop a process to define enforceable SLAs, start with the internal business units. These are easier to develop, as everyone has a shared business objective, and all disputes can be arbitrated by internal executives or team leaders.
Once the internal teams understand and are able to live with the metrics used to measure the SLAs, then this can be extended to vendors. The important part of this extension is that third-party SLA measurement organizations will need to become involved in this process.
Some would say that I am tooting my own horn by advocating the use of these third-party measurement organizations, as I have worked for two of the leaders in this area. The need for a neutral third-party is crucial in this scenario; it would be like watching a soccer match (football for the enlightened among you) without the mediating influence of the referee.
If your organization is now considering implementing SLAs, then it is crucial that these agreements are relevant and enforceable. That way, both parties understand and will strive to meet easily measured and agreed upon goals, and understand that there are penalties for not delivering performance excellence.
Copyright © 2025 Performance Zen
Theme by Anders Noren — Up ↑