 |
Merchant Madness Testing MethodologyDuring the first half of this holiday shopping season, Gomez will be holding the 1st Annual Merchant Madness retail website performance tournament. This head to head competition will test the response time, availability, and consistency of 64 of the largest online retailers. Below you will find information regarding the methodology used for this competition. Testing Periods (Rounds)Testing has been broken down into 6 rounds. The competition begins October 29th, 2007 and concludes with a winner on November 27th, 2007. Dates for the round are below:
Frequency and Location of TestsFor the Merchant Madness tournament, Gomez will test the participant’s homepages from ten geographically dispersed co-location facilities across multiple network providers within the United States. Testing will be performed at regular 60–minute intervals 24 hours a day. Co–location facilities used to collect performance data are located in: Atlanta, GA (InterNAP), Ashburn, VA (AT&T), Boston, MA (AT&T); Chicago, IL (Qwest); Dallas, TX (Verizon MCI); Denver, CO (Verizon MCI), New York, NY (Sprint); Los Angeles, CA (Level 3); San Jose, CA (Verizon MCI); and Seattle, WA (Savvis). GPN Testing AgentGomez tests emulate a standard Web browser across an Internet backbone connection (T1 speed or faster). Data captured and displayed represents the network performance of the transferred data from the perspective of a visitor who is not drawing any page content from a browser or client-side caching server. Merchant Madness Competition ScoringDuring every testing period, each online retailer will be ranked (1 through 64) in the categories of availability, response time, and consistency. The winner of the head-to-head match-up will be the online retailer with the lowest cumulative ranking score. Total Score = Availability Rank + Response Time Rank + Consistency Rank In order to avoid a “forfeit” during the head-to-head match-up, retailers must maintain a minimum availability of 90% and an average response time under 10 seconds. Response TimeResponse Time metrics reflect the time elapsed while downloading a page (including all embedded objects, JavaScripting and cascading style sheets). Data points from each successful test run are collected from the multiple testing nodes around the country and are aggregated to formulate benchmarks. The resulting metric represents the average time required to execute a single page without factoring in user latency. Although reported test response time metrics are unattainable by real users accessing these applications, the metrics serve as performance standards by which companies can gauge site improvement and measure Internet business effectiveness and customer experience. AvailabilityAvailability measures the percentage of completed transactions. Availability rates are dependent upon a number of variables including availability of the website, ability to process a transaction without error and ability to complete each step of the transaction within a reasonable 60-second time frame. Availability rates do not include server side errors or technical errors incurred due to scripting problems. These errors do occur from time to time and are reflected with a n/a (not available or not applicable) message on the benchmark. ConsistencyConsistency measures the standard deviation of the response time of completed transactions. The goal is to have a low number (in seconds) which indicates how consistent a web site delivers the business process. Usually a high consistency (in seconds) number indicates poor website geographic Internet connectivity or an under-powered technical architecture. |
