by

Suckkee Lee

Lecturer

Department of Advertising

School of Business Administration

Dongguk University

Seoul, Korea

Phone: (02) 671-7208

Email: suckkee@netsgo.com

and

John D. Leckenby

Everett D. Collier Centennial Chair in Communication

john.leckenby@mail.utexas.edu

http://www.utexas.edu/coc/admedium/

Department of Advertising

College of Communication

The University of Texas at Austin

Austin, Texas 78712

September 1998

forthcoming

Impact of measurement periods on website rankings is investigated using the data collected by a user-centric method. Their impact on the estimation of website traffic is also investigated. The findings indicate that site reach increases by approximately 71% from one day to one week and one week to four weeks, and site frequency increases by 55% from one week to one month. Site rankings are found affected not by the length of measurement periods but by the sorting criterion.

As the presence of the World Wide Web, hereinafter simply referred to as the Web, increases in our society, it has drawn much attention from advertisers, advertising agencies as well as academicians (Advertising Research Foundation 1995, Allen 1996, Bhatia 1997, Berthon et al. 1996a, Berthon et al. 1996b, Carter 1995, Cohen 1996, Danner 1997, Dolinar 1995, Economist 1997, FIND/SVP 1998, Gattuso 1995, Hoffman and Novak 1996, Lebow 1995, Murphy 1996, Randall 1997, Robello 1996). Accordingly, there are many research companies that provide a variety of analyses about the Web such as Nielsen/Ipro (at URL: http://www.ipro.com), Media Metrix, the PC Meter Company (URL: http://www. mediametrix.com), RelevantKnowledge (URL: http://www. relevantknowledge.com) and NetCount (URL: http://www.netcount.com), just to name a few. There are also industry-wide joint organizations such as the Coalition for Advertising Supported Information and Entertainment (CASIE at URL: http://www. commercepark.com/AAAA/bc/casie) and the Internet Advertising Bureau (IAB at URL: http://www.iab.net) which seek to provide a united front of the Web research industry to advertisers. Despite all these research efforts by various organizations, there is no industry-wide standard for Web traffic analysis, and a consensus as to how to measure the Web traffic is yet to be reached due to the technical and methodological issues (see, for example, Bruner 1997, Taylor 1997 and Haar 1997 for discussions of technical issues; Bhatia 1997, Schmetterer 1997 and Quistgaard 1997 for those of methodological issues).

The core of the methodological issues in traffic analysis of websites can be roughly categorized as what to measure and how to measure it. The former is related, in essence, to the analysis unit of site traffic such as hits, sessions, visits, pages, views and impressions (see Zeff and Aronson 1998 for a detailed discussion about the site-centric units of analysis) and the latter is related to issues such as site-centric v. user-centric approaches, cookies and on-line registration (see Lee 1998 for a detailed discussion about user-centric measurement methods). This paper explores one of the issues related to the latter category of how to measure the site traffic: the impact of measurement periods on site rankings and traffic estimation.

First, a distinction between measurement and estimation is made here to minimize possible confusion . The term "measurement" is used in this paper to mean to count the sampled data. The purpose of measurement is to estimate the size of the population from which the sampled data come. Therefore, one can say that measurement is to sample what estimation is to population. Since it is impossible to actually measure the true count of the whole Web traffic, the only way to assess the whole Web traffic is to estimate by counting sampled data and project the results of the measurement into the Web traffic.

Implications of the length of the measurement period in analyzing site traffic are such that, depending on it, estimations of site traffic-reach and frequency, if a user-centric approach is used-may differ significantly (see Hong and Leckenby 1996 and 1997 for the reach and frequency estimation models for websites; Danaher 1988, Leckenby 1986, Leckenby and Ju 1990 and Rust 1986 for further discussion of the reach and frequency estimation models in traditional media; Pasadeos et al. 1997 for a historical citation analysis of media planning in general). For example, traffic volume in some websites may fluctuate heavily from week to week or from weekdays to weekends while that of other sites may stay rather stable. Although the smallest unit used in this study is one day, it is possible that traffic volume of some sites may vary significantly from morning hours to afternoon hours to evening hours just like the audience flow in TV and radio.

More importantly, the length of measurement period has a direct influence on the advertisers' and media planners' decision. Prices for placing ads in a website these days is calculated in essence based on two units: 1) how many pages/views/impressions or unique users the website serves 2) for a certain length of period of time-mostly one day, one week or one month. To make a decision regarding the second unit, advertisers and media planners needs to know which advertising period would be the most efficient way to invest their advertising dollars.

Also, rankings of websites too may differ significantly depending on which criterion and/or which measurement period is used in sorting the site rankings. Discrepancies between the lists of the so-called "Top Websites", for instance, "Top 25 Sites" of RelevantKnowledge, "Top 50 Websites" of PC Meter, "100 Hot Websites" of Web21 (at URL: http://www.100hot.com) and "100 Most Popular Websites" of PC Magazine (at URL: http://www.pcmag.com/ special/web100), are such examples resulting from different ranking standards and/or different measurement periods used by individual companies. With these two issues in mind, the purpose of this study is to investigate the impact of different measurement periods on 1) site traffic, i.e., reach and frequency of websites and 2) site rankings sorted by different criteria in different measurement periods.

There are two issues to be investigated in the study: levels of site reach and site frequency in different measurement periods and site rankings sorted by different standards in different measurement periods. The first issue taps the possibility of any patterns in the flow of site traffic as the measurement period lengthens from one day to one week to one month. One implicit assumption behind the investigation of this issue is that there is a positive relation between site traffic and measurement period. In other words, the assumption is that site traffic grows as the measurement period gets longer. The second issue is about what factor(s), including the length of measurement period, affects the order of website rankings. The second issue is investigated first.

In order to rank sites, a standard or criterion is needed so that sites can be sorted in a descending order. Although other alternatives could be imagined for the sorting standard, it is assumed in this study that sites are ranked by the level of their reach or that of their frequency within a certain length of measurement period. First, in order to test the impact of different measurement periods on site rankings, the same standard, either reach or frequency, is applied, and the resulting rankings of websites are compared. Therefore, the first hypothesis (H1) is stated as follows:

H1: Rankings in different measurement periods will not differ if sorted by the same standard, i.e., either by site reach or site frequency.

The underlying rationale for H1 is that, provided the same standard is used in sorting the sites, the sites which get ranked high or low will also get ranked similarly regardless of the length of measurement periods. Sites ranked high in a month, for example, are likely to be ranked high in a week if the same standard is used for the rankings.

In the second hypothesis (H2), the impact of different sorting standards within the same measurement periods is explored. That is, sites are ranked by their reach and frequency respectively within the same measurement period-daily, weekly or monthly-and the agreement between two different rankings in the same measurement period is checked. Since it is quite plausible to hypothesize that there are two types of sites-sites that many people come to initially but return to less frequently and sites that not as many people come to initially but that enjoy a high rate of repeat visits from their audience, H2 is stated as follows:

H2: Rankings in same measurement periods will differ if sorted by different standards, i.e., by reach and frequency.

Regarding the impact of measurement period on site traffic, two research questions are raised. Provided that the assumption of positive relation between measurement period and site traffic is valid, the focus of two research questions is to see if there is any discernible pattern between the two. Since site traffic-if measured by a user-centric method-can be represented in two ways, reach and frequency, two research questions (Q1 and Q2) are posed.

Q1) What is the relationship between site reach and measurement period? (other than the positive one assumed)

Q2) What is the relationship between site frequency and measurement period? (other than the positive one assumed)

To test the hypotheses and answer the research questions above, the following set of definitions and equations is prepared for calculation and analysis of data. First, basic concepts such as "visit", "unique visitor" and "total traffic" of a site are defined, and based on them, site reach and frequency are defined.

• Visit: an occurrence of URL request for a site by one unique visitor.
• Unique Visitor: a singly identifiable person who visits a site(s) during a measurement period.
• Total Traffic (of a site): sum of all visits (to a site) during a measurement period.
• Reach (of a site): percentage of unique visitors who visited a website at least once during a measurement period. Therefore, site reach is an indication of how many different people visit a site during a measurement period.
• Frequency (of a site): average number of times of visits among those who visited a site during a measurement period. Therefore, site frequency is an indication of how many times people visit a site repeatedly during a measurement period.

From the previous definitions, the following set of equations is derived, i.e., if we let N denote the total number of unique visitors who have shown any web-browsing activities to any sites during a measurement period, then

and

therefore,

Where

R stands for reach; F for frequency; UV for unique visitor to a site; TT for total traffic of a site.

As is seen in the last equation above, the product of reach and frequency becomes total traffic (TT) of a site-if the percent variable, 100/N, is ignored for a moment-which is equivalent to the GRP (Gross Ratings Point) in the traditional media planning (see Beville 1988, Brown 1994 and Webster and Lichty 1991 for further discussions of media planning in traditional media).

A data set was obtained from RelevantKnowledge, a Web research firm in Atlanta, GA. The company adopts a user-centric tracking system by measuring activities of Web usage among its panel participants. Its definition of the universe of Web users is all people age 12 or older in the U.S. who have used the World Wide Web at least one time within the last month at home, work or college with Mac or PC with Windows 3.1, 95 and NT operating systems.

The initial data set contained the records of 744 unique visitors who were measured on a daily basis for one month-from September 29, 1997 to October 26, 1997. Those 744 unique visitors visited the top 99 sites, producing 13,039 records or visits. From these 13,039 records, only the records that related to the top 25 sites were extracted for analysis of the study. The initial sorting criterion that was used to extract the top 25 sites was the site reach for the measurement period above, i.e., monthly reach. The list of the top 25 sites and their reach for the monthly period are presented in Table 1.

Table 2

describes the summary of the extracted data and the demographics of the panel participants.

One drawback of the data should be noted here. Although the obtained data tells whether or not a unique visitor went to a site on a particular day, it does not tell how many times s/he visited the site in that day. In other words, it can be known from the data how many different sites a user visited on a particular day but not how many times he visited the same site on that day. Nor can it be known how many pages of a site he viewed. Therefore, the number of visits to the same site by a unique visitor in one day cannot be known from the data, and due to the lack of this information, daily frequency of sites cannot be calculated. However, the number of times he visits the same site during a longer measurement period, e.g., a week or four weeks, can be calculated from the data by counting the number of days of his visits to the site.

1. Hypotheses Testing

Since the data set was recorded on a daily basis during the month of October 1997, the records of the data are sliced into smaller chunks of daily and weekly data sets. Therefore, there are 3 measurement periods: daily, weekly and monthly periods. First, based on these 3 measurement periods, daily reach of the top 25 sites, averaged over 28 days, is tabulated. Second, weekly reach and frequency, averaged over 4 weeks, are tabulated. Third, monthly reach and frequency are tabulated (see Table 3). Finally, sites are sorted by each category-reach and frequency-in different measurement periods.

To test the impact of measurement periods on site rankings by one same criterion, Kendall's tau for rank-order correlation, both of which are an indication of "concordance" between different rankings (Hays 1988), are used on the site rankings which are sorted by the level of either reach or frequency for different measurement periods. Table 4 is the pairwise test results where the reach criterion is used for 3 different measurement periods and the frequency criterion is used for 2 different measurement periods. The overall concordance among the 3 rankings (Kendall's W) in the reach criterion is 93.6.

Table 4 shows that the rankings sorted by the same standard-either reach or frequency-for different measurement periods "concur" significantly with each other. Rankings in different measurement periods by the reach criterion concur, on average, 77.8% with each other, and those by the frequency criterion concur 94.7%. To state it simply, the reach rankings agree roughly 8 times out of 10, and the frequency rankings agree nearly 9.5 times out of 10. This agreement level is deemed to be safe to confirm H1.

In order to find out the impact of different sorting criterion, site rankings are compared by Kendall's tau while holding the measurement period constant. But be reminded that, as mentioned earlier, comparison between the rankings by daily reach and frequency cannot be made due to the lack of information about the daily frequency of the sites. Comparison of the rankings by the reach and frequency criteria in 2 different measurement periods-weekly and monthly-is shown in Table 5.

Overall agreement between the rankings by different standards in the same measurement period seems, judging from the concordance levels of .34 and .193 in Table 5, very weak. As for the weekly rankings, the agreement between the two rankings is tenuous (34%) although it is significant at .05 level. The agreement in the monthly period deteriorates further to 19.3% in concordance and it is not even a significant one. In short, the agreement between the rankings by different standards in the same measurement period is quite low or decent at best, supporting H2.

Since both H1 and H2 are supported by the data, a general conclusion can be drawn regarding the impact of the measurement period on site rankings: measurement periods do not affect the order of site rankings. Stated differently, as long as the same sorting criterion is applied, site rankings are accurate and fair regardless of the length of the measurement period. The deciding factor on site rankings is the employed criterion used for the rankings, which supports the underlying rationale of H2 that some sites get higher rankings by the reach criterion while other sites get more favorable rankings by the frequency criterion.

2. Assumption Checking

The first research question (Q1) of the study assumes that site reach increases as the measurement periods gets lengthened. The validity of this assumption is checked first before proceeding to the investigation of the more detailed relation between the two. Since the same 25 sites are measured repeatedly, a repeated measures analysis (Stevens 1992) is run on 3 levels of reach for different measurement periods. This is to establish first that there is a significant overall difference among the 3 levels of reach. The ANOVA table of the analysis is presented in Table 6. Since there is a significant overall difference (p<.00) among the 3 levels of reach depending on the measurement period, the following task is to compare 3 pairs of reach-daily-weekly, weekly-monthly, and monthly-daily combinations-so as to find out which pair(s) contributes to the overall difference. Paired t tests are run on the 3 combinations. Descriptive statistics of the combinations are presented at the bottom line of Table 3, and the test result is presented in Table 7.

Table 7 shows that all three pairs of reach show a significant difference depending on the measurement period, contributing to the overall difference. This confirms the validity of the assumption for Q1 and warrants an exploration of Q1.

The same assumption-checking is done on the second research question (Q2) of the study that as the measurement period gets longer, site frequency increases. Since there are only two frequency levels available-weekly and monthly, one paired t test is used to see if the two levels of frequency differ depending on measurement periods. Descriptive statistics of the pair are at the bottom line in Table 3, and the test result is presented in Table 8.

From Table 8 it can be observed that the same pattern of relationship occurs again between length of measurement period and level of site frequency as it does between site reach and measurement period. That is, although the level of frequency becomes significantly higher as the measurement period becomes longer, its margin of increase can not be known at this point.

As was confirmed in the assumption checking processes above, different measurement periods generate significantly different levels of site reach and frequency. However, the margin of the increase in both reach and frequency is not equivalent to that of the measurement period. This is to say that, although we know that site traffic increases as the measurement period lengthens, we do not know the ratio between the two. To put the research questions of the study in practical terms for media planners and advertisers who have to decide how long they will put their ads in a website, if the measurement period lengthens from a day to a week, will the site reach and frequency increase seven times too? If a website charges one dollar for placement of a banner ad for one week and four dollars for one month, is that pricing system a reasonable one? The answers to these questions are discussed in the following section.

3. Reach in Different Measurement Periods

Since assumptions for Q1 and Q2 have been validated, relationships between the levels of reach and frequency for different measurement periods merit a further investigation. Based on Table 3 where reach and frequency of each site in different measurement periods are presented, the relationship between site reach and measurement periods is analyzed first.

A cursory examination of the reach part in Table 3 enables one to sense the validity of the assumption for Q1, but a more exact relationship between reach and measurement periods emerges from Figure 1 which shows the ratio between three pairs of reach in different measurement periods.

The first pattern that jumps out from Figure 1 is that all three lines of ratio resemble one another very closely in their shapes. This pattern applies across all sites with notable exceptions of Site# 4 (att.net) and 11 (hotmail.com) where reach remains largely unchanged regardless of measurement periods. Further investigation about why the two sites show such a peculiar characteristic is beyond the scope of this study, but this first pattern clearly indicates that there is a discernible relationship between reach and three measurement periods.

The second pattern that can be detected from Figure 1 is a very similar flow of all 3 ratio lines, particularly between the monthly-weekly pair and the weekly-daily pair. In other words, increase in reach from one-day period to one-week period is almost equivalent to that from one-week period to monthly period. The wider fluctuation in the monthly-daily pair is merely the result of aggregating the other two lines.

The third pattern from Figure 1 is that, if the two exception sites-att.net and hotmail.com-are disregarded for a moment, variance in the ratio lines of the monthly-weekly and weekly-daily pairs are so stable that rough estimations of the increase in reach from one day to one week and from one week to one month become a feasible task. As can be seen in Table 9, the average ratio in the monthly-weekly pair and the weekly-daily pair is very close to each other (1.72 and 1.70 respectively), and so are their standard errors (.06 and .07). More importantly, the size of standard errors remains relatively small, making a rather stable estimation possible. Simply put, a projection that site reach will increase by 71% from a day to a week and from a week to a month would not miss the mark by much.

4. Frequency in Different Measurement Periods

The same procedure as in the previous analysis of reach and measurement periods is used again to find out the relationship between frequency and measurement periods. However, unlike in the reach and measurement periods, there is only one ratio line due to the lack of daily frequency. Figure 2 shows the ratio line between the weekly and monthly frequencies.

As in reach, Site# 4 and 11 show again an atypical pattern of frequency. Other than the two sites, the frequency ratio line for the other 23 sites remains mostly between 1.2 and 2. Although there are no other ratio lines to be compared against, the same observation can be made again regarding the relationship between frequency and measurement periods, i.e., standard error of the ratio line is small (.05) so that a stable estimation about the increase in frequency from a weekly period to a monthly period-55% on average (see Table 10)-is not out of the question.

Two conclusions can be drawn from the analyses about site reach, frequency and rankings in different measurement periods. First, as was assumed for the research questions 1 and 2, site reach and site frequency increases significantly as the measurement periods lengthen. But their rate of increase is not equivalent. In regard to reach, the rate of increase is about 71% with .06 and .07 standard errors as the measurement period increases from one day to one week and from one week to one month respectively. In regard to frequency, the rate of increase is about 55% with .05 standard error as the measurement period increases from one week to one month. Although the margin of standard errors-.06 and .07 in reach and .05 in frequency-seems to the authors tolerable enough to make stable projections about them, it is open to judgment of individual readers of this study.

Findings of the ratio lines in Figure 1 and 2 are more relevant to advertisers and media planners than researchers and academicians because they are the ones who pay for ads by the length of advertising period. Although this study takes a user-centric approach for assessment of site traffic, the length of advertising period is an issue of interest to both advertisers and this study. By the same token, the first finding of this study can also be of help to website owners when they price their websites.

The second conclusion of the study is about site rankings. In general, site rankings are not affected by the length of measurement periods as long as one common standard is applied in ranking the websites. Conversely speaking, it is the standard that decides the order of rankings. Which standard-reach, frequency or something else-is an effective one is a matter each advertiser should decide for himself before he buys spaces for his ads in websites. The att.net and hotmail.com sites where the audience shows a little different pattern of visit, for example, would be a good place for advertisers who seek to increase the frequency of their ads to a "small but loyal" audience. Likewise, the cnn.com, usatoday.com and sportszone.com sites belong to the same category as the att.net and hotmail.com sites while sites such as geocities.com, infoseek.com and lycos.com would be a reverse example. Certainly, the so-called "Big Three" sites-microsoft.com, netscape.com and yahoo.com-would be the ideal sites for the advertisers who want very high reach and high frequency for their ads.

Table 1

Rankings of the Top 25 Sites and Monthly Reach

Table 2

Summary of Data and Demographics of Panel Participants

Demographics

Table 3

Site Reach and Frequency in Different Measurement Periods

Table 4

Pairwise Concordance (Kendall's tau-b) between Rankings by Reach or Frequency in Different Measurement Periods (N=25)

Table 5

Pairwise Concordance (Kendall's tau-b) between Rankings by Reach and Frequency in Weekly and Monthly Measurement Periods (N=25)

Table 6

ANOVA Table for Repeated Measures Analysis

Table 7

Results of Paired t Tests for Reach

Table 8

Results of Paired t Tests for Frequency

Table 9

Descriptive Statistics of Ratio of Reach (N=23)

Table 10

Descriptive Statistics of Ratio of Frequency

Figure 1

Ratio of Reach in Different Measurement Periods

Figure 2

Ratio of Frequency in Different Measurement Periods