October 1999                                                                                                                 

Scott H. Irwin, Thomas E. Jackson and Darrel L. Good[*]

Copyright 1999 by Scott H. Irwin, Thomas E. Jackson and Darrel L. Good. All rights reserved. Readers may make verbatim copies of this document for non-commercial purposes by any means, provided that this copyright notice appears on all such copies.

DISCLAIMER

The advisory service marketing recommendations used in this research represent the best efforts of the AgMAS Project staff to accurately and fairly interpret the information made available by each advisory program.  In cases where a recommendation is vague or unclear, some judgment is exercised as to whether or not to include that particular recommendation or how to implement the recommendation.  Given that some recommendations are subject to interpretation, the possibility is acknowledged that the AgMAS track record of recommendations for a given program may differ from that stated by the advisory service, or from that recorded by another subscriber.  In addition, the net advisory prices presented in this report may differ substantially from those computed by an advisory service or another subscriber due to differences in simulation assumptions, particularly with respect to the geographic location of production, cash and forward contract prices, expected and actual yields, carrying charges and government programs

The purpose of this paper is to address two basic performance questions for market advisory services: 1) Do market advisory services, on average, outperform an appropriate market benchmark? and 2) Do market advisory services exhibit persistence in their performance from year-to-year?  Data on corn and soybean net price received for advisory services, as reported by the AgMAS Project, are available for the 1995, 1996 and 1997 marketing years.  Performance test results suggest that, on average, market advisory services exhibit a small ability to "beat the market" for the 1995 through 1997 corn and soybean crops.  This conclusion is somewhat sensitive to the type of performance test and market benchmark considered.  The predictability results provide little evidence that future advisory service pricing performance can be predicted from past performance.  When services are grouped by performance quantile, some evidence of predictability is found for the poorest performing services, but not for top performing services.

Do Agricultural Market Advisory Services Beat the Market? Evidence from the Corn and Soybean Markets Over 1995-1997

Price risk management is an important business activity for US grain farmers.  Using a survey of large-scale, midwestern grain farmers, Patrick and Ullerich (1996) report that price variability is the highest rated source of risk by crop farmers.  Given the dramatic fluctuations of grain prices in recent years, it is likely that price variability will continue to be a major source of risk for farmers.

Farmers view market advisory services as a significant source of market information and advice in their quest to manage price risks associated with grain marketing.  In a rating of seventeen risk management information sources, Patrick and Ullerich (1996) report that the rank of computerized information services and market advisors is surpassed only by farm records.  Patrick, Musser, and Eckman (1998) indicate that 35 and 38 percent of large-scale, midwestern grain farmers used marketing consultants in 1993 and 1994, respectively.  Schroeder, Parcell, Kastens and Dhuyvetter (1998) surveyed Kansas crop farmers and report that market advisory services and newsletters are the highest ranked source of information used to formulate price expectations.  It is interesting to note that advisory services outranked even futures markets in this latter survey.

Given the high value that farmers place upon market advisory services, it is somewhat surprising that only two academic studies investigate the pricing performance of advisory services.[1]  The dearth of studies seems even more anomalous in light of the large number of studies on grain marketing strategies.[2]  The lack of studies on market advisory services is most likely due to the difficulty in obtaining data on the stream of recommendations provided by services.

Gehrt and Good (1993) analyze the performance of five advisory services for corn and soybeans over 1985 though 1989.  Assuming a representative producer follows the hedging and cash market recommendations for each advisory service, a net price received for each year is computed and compared to a benchmark price.  They generally find that corn and soybean farmers obtained a higher price by following the marketing recommendations of advisory services.  Martines-Filho (1996) examines the pre-harvest corn and soybean marketing recommendations of six market advisory services over 1991 through 1994.  He computes the harvest time revenue that results from a representative farmer following the pre-harvest futures and options hedging recommendations and selling 100 percent of production at harvest.  Average advisory service revenue over the four years is larger than benchmark revenue for both corn and soybeans.

While a useful starting point, the two previous studies have important limitations.  First, the sample of advisory services is quite small, with the largest sample including only six advisory services.  Second, the results may be biased due to the nature of the sample selection process.  The literature on the performance of mutual funds and investment newsletters highlights the sample selection biases that plague many performance results (e.g. Brown, Goetzmann, Ibbotson, and Ross, 1992; Jaffe and Mahoney, 1999; Metrick, 1999).  The most relevant bias for previous studies of market advisory services is survivorship bias, which results from tracking only advisory services that remain in business at the end of a sample period.

The previous discussion suggests the academic literature provides farmers with little basis for evaluating and selecting advisory services.  In 1994, the Agricultural Market Advisory Service (AgMAS) Project was initiated, with the goal of providing unbiased and rigorous evaluation of market advisory services for farmers.  Since its inception, the AgMAS Project has collected marketing recommendations for about 25 market advisory programs.  The AgMAS Project subscribes to all of the services that are followed, and as a result, "real-time" recommendations are obtained.  This prevents the data from being subject to survivorship bias.

After the stream of recommendations is collected for a given commodity in a particular marketing year, the net price that would have been received by a producer that precisely follows the set of marketing recommendations is computed.  This net price is the weighted average of the cash sale price plus or minus gains/losses associated with futures and options transactions.  Brokerage costs are accounted for, as are the costs of storing any portion of the crop beyond harvest.  So far, the AgMAS Project has reported corn and soybean results for the 1995, 1996 and 1997 marketing years. (Good, Irwin, Jackson, and Price, 1997; Jackson, Irwin, and Good, 1998; Jackson, Irwin, and Good, 1999).

The annual AgMAS comparison of net price received for advisory services provides  important information that farmers can use in selecting a service.  However, the comparisons to date are descriptive only and do not rigorously address the central questions regarding pricing performance.  Following the literature on mutual fund and investment newsletter performance (e.g. Jaffe and Mahoney, 1999), two basic questions need to be answered: 1) Do market advisory services, on average, outperform an appropriate market benchmark? and 2) Do market advisory services exhibit persistence in their performance from year-to-year? 

The purpose of this report is to address the previous two questions for corn and soybeans using the net advisory prices reported by the AgMAS Project for the 1995, 1996 and 1997 marketing years.  At least 21 advisory services are included in the evaluations for each commodity and marketing year.  While the sample of advisory services is non-random, it is constructed to be generally representative of the majority of advisory services offered to farmers.  The availability of only three marketing years is a limitation of the analysis, but the time period considered does include years of rapidly increasing and decreasing corn and soybean prices.

 The tests used to determine average performance of market advisory services and predictability of performance through time have been widely applied in the financial literature (e.g. Elton, Gruber, and Rentzler, 1987; Lakonishok, Shleifer and Vishny, 1992; Irwin, Zulauf, and Ward, 1994;  Jaffe and Mahoney, 1999; Metrick, 1999).  Tests of performance relative to a benchmark are based on the proportion of services exceeding the benchmark price and the average percentage difference between the net price of services and the benchmark price.  Tests of predictability are based on the year-to-year correlation of advisory service ranks, prices and percentage differences from the benchmark.  In addition, predictability is examined for advisory services in different performance quantiles.

The market advisory services included in this evaluation do not comprise the population of market advisory services available to farmers.  The included services also are not a random sample of the population of market advisory services.  Neither approach is feasible because no public agency or trade group assembles a list of advisory services that could be considered the "population."  Furthermore, there is not a generally agreed upon definition of an agricultural market advisory service.  To assemble a sample of services for the AgMAS Project, criteria are developed to define an agricultural market advisory service and a list of services is assembled.

The first criterion used to identify services is that a service has to provide marketing advice to farmers. Some of the services tracked by the AgMAS Project do provide speculative trading advice, but that advice must be clearly differentiated from marketing advice to farmers for the service to be included.  The terms "speculative" trading of futures and options versus the use of futures and options for "hedging" purposes are used for identification purposes only.  A discussion of what types of futures and options trading activities constitute hedging, as opposed to speculating, is not considered.

The second criterion is that specific advice must be given for making cash sales of the commodity, in addition to any futures or options hedging activities.  In fact, some marketing programs evaluated by the AgMAS Project do not make any futures and options recommendations.  However, marketing programs that make futures and options hedging recommendations, but fail to clearly state when cash sales should be made, or the amount to be sold, are not considered.

The original sample of market advisory services that met the two criteria were drawn from the list of  "Premium Services" available from the two major agricultural satellite networks, Data Transmission Network (DTN) and FarmDayta in the summer of 1994.[3]^{, [4]}  While the list of advisory services available from these networks was by no means exhaustive, it did have the considerable merit of meeting a market test.  Presumably, the services offered by the networks were those most in demand by farm subscribers to the networks.  In addition, the list of available services was cross-checked with other farm publications to confirm that widely-followed advisory firms were included in the sample.  It seems reasonable to argue that the resulting sample of services was (and remains) generally representative of the majority of advisory services available to farmers.

The original sample for 1995 includes 25 market advisory programs for both corn and soybeans.  For a variety of reasons, deletions and additions to the original sample occur over time.[5]  In 1996, the total number of advisory programs is 26 for corn and 24 for soybeans, while in 1997 the total is 23 for corn and 21 for soybeans.  The term “advisory program” is used because several advisory services have more than one distinct marketing program.[6]  A directory of the advisory services included in the study can be found at the AgMAS Project website (http://www.aces.uiuc.edu/~agmas/).

 As mentioned earlier, sample selection biases may plague advisory service databases.  The first form is survival bias, which occurs if only advisory services that remain in business at the end of a given period are included in the sample.  Survival bias significantly biases measures of performance upwards since "survivors" typically have higher performance than "non-survivors" (Brown, Goetzmann, Ibbotson, and Ross, 1992).  This form of bias should not be present in the AgMAS database of advisory services because all services ever tracked are included in the sample.  The second and more subtle form of bias is hindsight bias, which occurs if data from prior periods are "back-filled" at the point in time when an advisory service is added to the database.  Statistically, this has the same effect as survivorship bias because data from surviving advisory services is back-filled.  This form of bias should not be present in the AgMAS database because recommendations are not back-filled when an advisory service is added.  Instead, recommendations are collected only for the marketing year after a decision has been made to add an advisory service to the database.

The actual daily process of collecting recommendations for the sample of advisory services begins with the purchase of subscriptions to each of the services.  Staff members of the AgMAS Project read the information provided by each advisory service on a daily basis.  The information is received electronically, via DTN, web sites or email.  For the services that provide two daily updates, typically in the morning and at noon, information is read in the morning and afternoon.  In this way, the actions of a farmer-subscriber are simulated in “real-time.”

The recommendations of each advisory service are recorded separately.  Some advisory services offer two or more distinct marketing programs.  This typically takes the form of one set of advice for marketers who are willing to use futures and options (although futures and options are not always used), and a separate set of advice for farmers who only wish to make cash sales.[7]  In this situation, both strategies are recorded and treated as distinct strategies to be evaluated.[8]

Several procedures are used to check the recorded recommendations for accuracy and completeness.  Whenever possible, recorded recommendations are cross-checked against later status reports provided by the relevant advisory service.  Also, at the completion of the marketing year, it is confirmed whether cash sales total exactly 100%, all futures positions are offset, and all options positions are offset or expire worthless.

At the end of a marketing year, all of the (filled) recommendations are aligned in chronological order.  The advice for a given marketing year is considered to be complete for each advisory program when cumulative cash sales of the commodity reach 100%, all open futures positions covering the crop are offset, all open option positions covering the crop are either offset or expired, and the advisory program discontinues giving advice for that crop year.  The returns to each recommendation are then calculated in order to arrive at a weighted-average net price that would be received by a producer who precisely follows the marketing advice (as recorded by the AgMAS Project).

In order to simulate a consistent and comparable set of results across the different advisory services, certain explicit assumptions are made.  These assumptions are intended to accurately depict marketing conditions for a representative, central-Illinois farm.  An overview of the simulation assumptions is presented below.  Complete details of the simulation assumptions can be found in Jackson, Irwin and Good (1999).

Marketing Window

A two-year marketing window, spanning September of the year before harvest through August of the year after harvest, is used in the analysis.  For example, the 1997 marketing window is September 1, 1996 through August 31, 1998.  The beginning date is selected because advisory services in the sample generally begin to make marketing recommendations around this date.  The ending date is selected to be consistent with the ending date for corn and soybean marketing years as defined by the US Department of Agriculture (USDA).  There are a few exceptions to the marketing window definition.  Several advisory programs have relatively small amounts (10% or less) of cash corn or soybeans unsold as of the end of a window.  One marketing program also began pre-harvest hedges prior to September 1, 1996.  In these cases, the actual sales recommendations on the indicated dates are recorded.

Prices

The cash price assigned to each cash sale recommendation is the central-Illinois closing, or overnight, bid.  The central-Illinois price is the mid-point of the range of bids by elevators in a 25-county area in central and east central-Illinois.  The bids are collected and reported by the Illinois Department of Agriculture.  The central-Illinois market also is used for cash-forward contract transactions.  Futures prices and options premia are Chicago Board of Trade quotes.

Quantity Sold

Since most of the advisory program recommendations are given in terms of the proportion of total production (e.g., “sell 5% of 1997 crop today”), some assumption must be made about the amount of production to be marketed.  For the purposes of this study, if the per-acre yield is assumed to be 100 bushels, then a recommendation to sell 5% of the corn crop translates into selling 5 bushels.  When all of the advice for the marketing year has been carried out, the final per-bushel selling price is the average price for each transaction weighted by the amount marketed in each transaction.

When making hedging or forward contracting decisions prior to harvest, the actual yield is unknown.  Hence, an assumption regarding the amount of expected production per acre is necessary to accurately reflect the returns to marketing advice.  Prior to harvest, the best estimate of the current year’s expected yield is a function of yield in previous years.  In this study, the assumed yield prior to harvest is based on a linear regression trend yield, while the actual reported yield is used from the harvest period forward.

Brokerage Costs

Brokerage costs are incurred when farmers open or lift positions in futures and options markets.  For the purposes of this study, it is assumed that brokerage costs are $50 per contract for a round-turn for futures transactions, and $30 per contract to enter or exit an options position.  Further, it is assumed that CBOT corn and soybean futures are used, and the contract size for each commodity is 5,000 bushels.  Therefore, per-bushel brokerage costs are 1 cent per bushel for a round-turn futures transaction and 0.6 cents per bushel for each options transaction.

Carrying Costs

An important element in assessing returns to an advisory program is the economic cost associated with storing grain instead of selling grain immediately at harvest.  The cost of storing grain after harvest (carrying costs) consists of two components: physical storage charges and the opportunity cost incurred by foregoing sales when the crop is harvested.  Physical storage charges can apply to off-farm (commercial) storage, on-farm storage, or some combination of the two.  Opportunity cost is the same regardless of the type of physical storage.

For the purposes of this study, it is assumed that all storage occurs off-farm at commercial sites.  Carrying costs are assigned beginning at the ending of the estimated ending points of the harvest windows.  Physical storage charges are assumed to be a flat 13 cents per bushel from the end of harvest through December 31.  After January 1, physical storage charges are assumed to be 2 cents per month (per bushel), with this charge pro-rated to the day when the cash sale is made.  The storage costs represent the typical storage charges quoted in a non-random telephone survey of Central-Illinois elevators.

The interest charge for storing grain is the interest rate compounded daily from the harvest mid-point to the date of sale.  The interest rate used is the average rate for all commercial agricultural loans for the fourth quarter of the harvest year and the first three quarters of the next calendar year as reported in the Agricultural Finance Databook published by the Board of Governors of the Federal Reserve Board.  This interest rate has been around 9% per year for the three years of this study.

In addition to the storage and interest costs, another charge is assigned to corn (but not soybeans).  This charge, referred to as a “shrink charge”, is commonly deducted by commercial elevators on “dry” corn that is delivered to the elevator to be stored, and reflects a charge for drying and volume reduction (shrinkage) which occurs in drying the corn from (typically) 15% to 14% moisture.  The charge for drying is a flat 2 cents per bushel, while the charge for volume reduction is 1.3% per bushel.  The charge for this volume reduction is calculated as 1.3% times the average harvest-time cash price for each marketing year.  For example, for the 1997 crop the harvest-time cash price was $2.65 per bushel, so the charge for volume reduction was 3.4 cents per bushel ($2.65*1.3%).

Simply comparing the net price received across advisory services will not answer the question of whether advisory services as a group enhance the income of farm subscribers.  Instead, a comparison to a benchmark price (or prices) is needed to evaluate the performance of advisory services relative to pricing opportunities offered by the market.  In the stock market, mutual funds are evaluated with respect to market benchmark performance criteria (e.g., Bodie, Kane, and Marcus, 1989).  These benchmarks typically are indexes of stock market returns over the period of evaluation, e.g., the Dow Jones Industrial Average and Standard and Poor’s 500.

The selection of appropriate benchmarks for advisory service performance evaluations is treated thoroughly in a recent report by Good, Irwin and Jackson (1998).  They argue that, conceptually, a useful benchmark should: 1) be simple to understand and to calculate; 2) represent the returns to a marketing strategy that could be implemented by farmers; 3) be directly comparable to the net advisory price received from following the recommendations of a market advisory service; 4) not be a function of the actual recommendations of the advisory services or of the actual marketing behavior of farmers, but rather should be external to their marketing activities; and 5) be stable, so that it represents the range of prices made available by the market throughout the marketing year instead of representing the price during a small segment of the marketing year.  The market benchmark price that Good, Irwin and Jackson argue is the most consistent with the above criteria is the average cash price for corn and soybeans over the entire marketing horizon.  The marketing window used in the AgMAS project for a given crop spans two calendar years, beginning on the first business day of September in the year prior to harvest, and extends through the last business day of August in the year after harvest.  As its name suggests, it is calculated as the average of the daily central-Illinois cash grain bids available for the two-year marketing window.  Pre-harvest cash prices represent cash-forward bids for harvest delivery in central-Illinois, while daily spot prices for central-Illinois are used for the post-harvest period.

Two adjustments are made to the daily cash prices to make the average cash price benchmark consistent with the calculated net advisory prices for each marketing program.  First, instead of taking the simple average of the daily prices, a weighted average price is calculated to account for changing yield expectations.  The daily weighting factors for pre-harvest prices are based on the calculated trend yield, while the weighting of the post-harvest prices is based on the actual reported yield for Central-Illinois.  The second adjustment to the daily cash prices is to adjust the post-harvest cash prices to a harvest equivalent by subtracting carrying charges.  The daily carrying charges are calculated in the same manner as those for the net advisory price. Complete details of the construction of this benchmark price can be found in Good, Irwin and Jackson (1998).

In order to test the sensitivity of performance results to the choice of market benchmark,  two alternative versions of the previous average cash price benchmark also are considered in the analysis.  The first alternative benchmark averages prices for the 20-month period starting in January of the year of harvest and ending in August of the year after harvest. The only difference between this alternative and the 24-month benchmark is the exclusion of the pre-harvest period previous to January.  Hence, this alternative benchmark places more weight on post-harvest prices than pre-harvest prices.  The second alternative benchmark averages prices only for a 12-month marketing year, and includes only post-harvest prices in the averaging process.[9]

Net price received for the sample of market advisory services for the 1995, 1996, and 1997 marketing years is reported in Tables 1 and 2.[10]^, [11]  Note that some of the marketing programs included in the table are not evaluated for all three years.  The three-year averages are calculated only for the 19 marketing programs that are evaluated for all three years.

As shown in Table 1, the average net advisory price for corn ranges from $2.32 per bushel in 1997 to $3.03 per bushel in 1995.  The three-year average for the 19 programs is $2.65 per bushel.  The range of average net advisory prices is large, with a low of $2.36 and a high of $3.03.  Not surprisingly, the range within the individual years is even more substantial.  The most dramatic example is 1995, where the minimum is $2.29 per bushel and the maximum is $3.90 per bushel!  Even in years with less market price volatility, such as 1997, the range in performance is just under $0.75 per bushel. 

The three alternative market benchmark prices for corn are shown at the bottom of Table 1.  Three-year averages of the market benchmarks differ by less than 10 cents per bushel.  However, this masks large differences within some of the years, particularly 1995.  These data suggest advisory service performance results for corn may be sensitive to the selected benchmark.  

The three-year results for soybeans are listed in Table 2.  The average net advisory price for soybeans ranges from $6.40 per bushel in 1997 to $7.27 per bushel in 1996.  The three-year average for the 19 programs is $6.73 per bushel.  Again, the range of average net advisory prices is large, with a low of $6.37 and a high of $7.27.  As with corn, the range within the individual years is even more substantial.  The most dramatic example is 1995, where the range in advisory prices exceeds two dollars per bushel.

Since many subscribers to market advisory services produce both corn and soybeans, it also is of interest to examine a combined measure of corn and soybean pricing performance for each market advisory program.  One way to aggregate the results is to calculate the per-acre revenues implied by the pricing performance results.[12]  The per-acre revenue for each commodity is found by multiplying the net advisory price for each market advisory program by the actual central-Illinois corn or soybean yield for each year.  A simple average of the two per acre revenues is then taken to reflect a farm that uses a 50/50 rotation of corn and soybeans.

Table 3 contains the combined corn and soybeans revenue results.  As with Tables 1 and 2, a three-year average is calculated only for programs that were included in the study for all three years.  In addition, market advisory programs that provide recommendations for corn but not soybeans (Ag Line by Doane hedge and Allendale futures & options) are not included.  The three-year average revenue for all 19 market advisory programs is $332 per acre.  The three-year average for the individual programs ranges from a low of $312 per acre to a high of $360 per acre. 

Two statistical tests are used to test the null hypothesis that average market advisory service pricing performance does not differ from that of the market benchmark.  The first test is  based on the proportion of services exceeding the benchmark price.  This test is considered because it is not influenced by extremely high or low advisory prices.  The second test is based on the average percentage difference between the net price of services and the benchmark price.  This test is useful because it takes into account the average magnitude of differences from the benchmark.

Independence of Observations

Before considering the statistical test results, an important issue needs to be explored that may have a substantial impact on the results.  The issue is whether the sample observations on net advisory price are independent, both within and across years.  The most likely form of dependence is positive correlation, which, if ignored, would cause sample standard deviation estimates across advisory services to be understated.  This in turn would cause the statistical significance of hypothesis test results to be overstated. 

There are several potential ways that independence could be violated in the sample of market advisory service prices.  One potential violation is positive correlation of corn pricing performance for a market advisory program in a given year with its soybean pricing performance in the same year.  In other words, do services that do well in corn also tend to do well in soybeans in the same year?  If so, statistical tests that pool pricing performance of services for corn and soybeans may overstate the significance of positive or negative performance because the standard deviation across the corn and soybean observations would be understated.

The correlation results for market advisory corn and soybean pricing performance within the same marketing year are summarized in Table 4.  Correlation in a given year is computed three ways.  First, the correlation of rank across corn and soybeans for a given year is computed. To do this, the rank of each advisory service with respect to the other services is calculated separately for corn and soybeans.  The services are ranked in descending order.  For example, the service with the highest net advisory price is ranked number 1, and the service with the lowest net advisory price is assigned a number equal to the total number of observations for that commodity in the given year.  The final step is to compute the correlation of the corn and soybean ranks.  Second, the simple correlation between the net advisory corn and soybean price levels is computed for a given year.  Third, the correlation of advisory service performance with respect to the 24-month market benchmark price is calculated.[13]  The “return” to market advice is calculated as the percentage difference between the net advisory price and the 24-month market benchmark price for the commodity.  A graphical view of the rank and return correlations is presented in Figure 1.

The results presented in Table 4 are similar across the different measures of correlation.  Significant positive correlation between corn and soybean pricing results is found in 1995 and 1997, but not for 1996.  This may be due to the fact that the price patterns for corn and soybeans were quite different for the 1996 crop year, while corn and soybean prices moved (generally) in the same direction during the 1995 and 1997 marketing years.  While market advisory programs do not make exactly the same recommendations for corn and soybeans in any given year, there often is a significantly positive correlation in their corn and soybean pricing performance.  This suggests it is inappropriate to pool separate corn and soybean pricing results when conducting statistical tests.

A second potential source of dependence is correlation of net advisory prices through time for a given service and commodity.  This form of correlation may exist due to persistence in the performance of advisory services through time (winners continue to win, losers continue to lose).  It may also exist due to the overlapping nature of the marketing years; each marketing year is two calendar years long, and each set of contiguous marketing years overlaps by one year.  If this correlation through time exists, it would be inappropriate to pool samples of net advisory prices across marketing years for the same reason as discussed above.  As will be shown in the following section, this form of correlation generally is quite low, and therefore, it is reasonable to pool net advisory prices across marketing years. 

A third potential source of dependence perhaps is less obvious.  It is possible that net advisory prices for a given commodity and marketing year are correlated because of the existence of similar programs offered by the same market advisory service.  For example,  AgriVisor offers four marketing programs, which may not differ substantially in outcomes due to similar methods of analysis and similar underlying strategies.  The potential impact of this form of correlation is examined by creating one net advisory price for each of the market advisory firms that offer multiple programs.[14]  A single price is computed by averaging net advisory prices across programs for a given year and commodity.  Pricing performance results are qualitatively similar to those using the full set of disaggregated advisory prices, suggesting that net prices of advisory programs for the same firm are uncorrelated or no more correlated than net prices from different firms.  Hence, use of net advisory prices by program in tests of market performance does not appear to be a substantive problem.

Test Statistics

A formal test of the null hypothesis that the proportion of advisory services "beating" the market benchmark is insignificant requires the specification of an appropriate test statistic.  Anderson, Sweeney and Williams  (1996) show that the sample estimator of the proportion, , is distributed binomially with an expected value of p and a standard error of, where p is the true value of the proportion in the population and n is the number of sample observations.  They also note that the sampling distribution of  is approximately normal so long as  and .  Since both conditions are met for all of the samples considered here, the normality approximation is invoked.  The form of the test statistic based on the above assumptions is , where p₀ is the assumed value of p under the null hypothesis.  The remaining issue is the expected proportion (p₀) under the null hypothesis.  The efficient market hypothesis (Fama, 1970) implies that the expected probability of “beating the market” is the same as the result of flipping a coin and showing heads, or 0.5.  Setting , the test statistic is .

             A formal test of the null hypothesis that the average percentage difference between the net price of services and the benchmark price is zero also requires the specification of an appropriate test statistic.  First, for a given marketing year and commodity, define the percentage difference for the i^th advisory service as , where NAP_i is the net advisory price for the i^th advisory service and BP is the market benchmark price for the same commodity and marketing year. The sampling distribution of  is well-known and does not need to be described in detail here.  The test statistic for a null hypothesis of zero average percentage difference is   where  is the estimated standard deviation of the differences across the n advisory services in the sample.  The t-statistic follows a t-distribution with n-1 degrees of freedom. 

As noted earlier, r_i can be thought of as the “return” to following the recommendations of a particular market advisory service.  This raises the question of whether the calculated “returns” are risk-adjusted.  If one is willing to assume that the average risk of advisory services is equal to risk of the market benchmark, then market advisory returns can be considered risk-adjusted returns.  This type of approach (risk-matching) is used frequently in studies of returns to strategies in financial markets (e.g. Ritter, 1991).  However, since it is difficult to test the appropriateness of this assumption over the short time period considered in this analysis, a risk-adjusted interpretation of advisory returns should be treated with a good bit of caution.

It is important to emphasize that the tests discussed in this section address the pricing performance of market advisory services as a group.  In other words, average pricing performance across all services is considered.  This is a different issue than the pricing performance of a particular advisory service.  It is possible that advisory services as a group fail to beat the market, yet at the same time there exist a small number of services that are exceptions to this outcome. In the stock market, this argument is often made with respect to the performance of the Fidelity Magellan Fund.  Testing whether an “exceptional” advisory service beats the market requires more data than is available for this study and different statistical methods (Marcus, 1990). 

Test Results

Table 5 reports results of the proportional test of corn pricing performance for each year and all three years pooled.  Individual year results are quite sensitive to the benchmark considered.  For example, the proportion of services above the 24-month benchmark price in 1995 is 0.72 and statistically significant, while the proportion of services above the 12-month benchmark is only 0.08.  This latter proportion is also statistically significant, but in the opposite direction, indicating significantly inferior performance.  Despite the variation across benchmarks for individual years, the overall proportions for the three years are similar across the benchmarks, ranging only from 0.51 to 0.59.  None of the three-year proportions are significantly different from 0.5 at the five- or ten-percent level, although the 12-month benchmark proportion is quite close to significance at the ten-percent level.

Table 6 shows the results of the proportional test of soybean pricing performance for each year and all three years pooled. Like corn, individual year results are sensitive to the benchmark considered.  The most dramatic contrast again can be found in 1995, where the proportion of services above the 24-month benchmark price is 0.84 and statistically significant, while the proportion of services above the 12-month benchmark is only 0.16.  The overall proportions for the three years range from 0.57 to 0.77.  Two of the three-year proportions (24-month and 20-month benchmarks) are significantly greater than 0.5 at the one-percent level.

Table 7 reports proportional test results for combined corn and soybean revenue.  Given the evidence of positive correlation between the pricing performance of advisory programs for corn and soybeans in the same year, it is inappropriate to simply pool the separate net price observations for corn and soybeans to test combined performance.  Instead, corn and soybean net prices are aggregated to form a single observation on per-acre revenue for each advisor and year, and then proportions are computed.  The per-acre combined revenues are those first presented in Table 3.  As would be expected, the proportions for revenue per acre fall between the proportions for corn and soybean net advisory prices and show a similar pattern of variation across the alternative benchmarks in a given year.  Combined corn and soybean performance for the entire three-year period is less variable across the benchmarks, with the proportion of programs above the benchmark ranging only from 0.60 to 0.64.  It is noteworthy that the three-year proportions are significantly above 0.5 for all three benchmarks.

 Results for the average return test of pricing performance are reported in Tables 8, 9 and 10.  Individual year and three-year average test results for corn, shown in Table 8, are qualitatively the same as the proportional test results. Point estimates of the three-year average returns range from –0.34 to 0.74 percent.  However, none of the three-year average corn returns are significantly different from zero.  Individual year and three-year average results for soybeans, reported in Table 9, are qualitatively similar to the proportional test results.  The only differences occur in 1997 for the 24-month and 20-month benchmarks, where significance is detected for average soybean returns but not the proportion of services above the market.  Point estimates of the three-year average soybean returns range from 0.71 to 3.00 percent, substantially higher than for corn.  Two of the three-year average soybean returns are significantly different from zero (24-month and 20-month benchmarks).  Results of the average return test for combined corn and soybean revenue, found in Table 10, differ the most from proportional test results.  Three-year average revenue returns are significant only for the 24-month benchmark, whereas three-year proportions are significant for all three benchmarks.  This divergence in results appears to be due to large, negative returns in some years (e.g. 1995, 12-month average benchmark) and relatively higher variation in returns as compared to proportions.  Point estimates of the three-year average revenue returns range from -0.30 to 1.84 percent, which, as expected, is between the ranges for corn and soybeans. 

In statistical terms, the pricing performance test results presented in this section are fairly clear.  Little or no evidence is found regarding the ability of market advisory services to consistently and significantly “beat the market” for corn.  There is substantial evidence that market advisory services consistently and significantly “beat the market” in soybeans.  When corn and soybean net advisory prices are combined into revenue per acre, some evidence also is found that market advisory services significantly outperform the market.  Tests results for revenue are the most sensitive to the type of test and benchmark considered.  Overall, the statistical results suggest that market advisory services have some ability to outperform broad market benchmarks.

Given the statistical results summarized above, a relevant question to ask is whether the pricing performance of advisory services also is economically significant.  While "economic significance" is a vague concept, it is important nonetheless.  Perhaps the best perspective on this question is gained by re-examining returns for corn and soybean revenue per acre.  Given the sensitivity of measured returns to the benchmark considered, the best point estimate of revenue returns probably is the simple average across the three benchmarks.  This “grand average” revenue return across all three marketing years is 0.74 percent, which translates into about $3 per acre above benchmark revenue.[15]  While this level of return is probably best characterized as “small,” it also appears to be non-trivial, particularly in comparison to the cost of the services.  Jackson, Irwin and Good (1999) report that the average cost of the services is $279 per year.  For a 1,000 acre corn and soybean farm, this translates into an average cost of only 28 cents per acre.  There are two important reasons to be cautious about concluding that advisory returns generate even a "small" level of economic significance: i) the results are based on a limited sample of years, and ii) returns are concentrated in only one market, soybeans.

The results of the analysis also have implications for the ongoing debate about market efficiency and risk management strategies in agriculture.  One view is that grain markets (cash, futures and options) are not efficient and, therefore, provide opportunities for farmers to systematically earn additional profits through marketing (e.g Wisner, Blue and Baldwin, 1998).  The other view is that grain markets are at least efficient with respect to the type of strategies available to farmers (e.g., Zulauf and Irwin, 1998).  Since the return of advisory services over 1995-1997 significantly exceeds transactions costs in some cases, including the cost of the services, the results potentially imply a rejection of market efficiency in the sense of Grossman and Stiglitz (1980).[16]  A firm conclusion cannot be reached due to the uncertainties pointed out with respect to economic significance.  In addition, there is uncertainty about the appropriate adjustment for risk or a complete accounting for the costs of implementing advisory service recommendations.  It may be that important costs are ignored, such as search costs, monitoring costs and related management costs.  Nevertheless, the performance results suggest market advisory services, at least to a modest extent, have some access to information not available to other market participants and/or superior analytical skills.  

Finally, it is interesting to compare the pricing performance results for market advisory services to that of other investment professionals.  According to Morningstar Reports, only 16% of active mutual fund managers beat the returns to a broad stock market average over the last decade (Clements, 1999).  By comparison, the performance of agricultural market advisory services is quite strong, with about half of the services beating the market in corn and about two-thirds beating the market in soybeans.  This divergence may simply reflect a unique time period in corn and soybean markets, relatively less efficient commodity markets, the skillfulness of advisory services, or an inappropriate adjustment for advisory service risk.  Determining which explanation is correct will be an important subject for future research as more data on market advisory performance becomes available. 

 Even if, as a group, advisory services generate positive returns, there is a wide range in performance for any given year.  For example, soybean net advisory prices for 1995 vary from $5.71 per bushel to $7.94 per bushel.  While this example probably is the most dramatic, the variation across advisors in other cases is substantial.  This raises the important question of the predictability of advisory service performance from year-to-year.  In other words, is past performance indicative of future results?  This issue is addressed two ways: i) by calculating correlation coefficients for measures of advisory service performance across adjacent marketing years, and ii) determining the average performance for services ranked by quantiles in a year subsequent to the initial year.  The testing procedures have been widely applied in studies of financial investment performance (Elton, Gruber, and Rentzler, 1987; Irwin, Zulauf and Ward, 1994; Lakonishok, Shleifer and Vishny, 1992).  Recent analysis by Brorsen and Townsend (1998) indicates these methods are reasonably powerful in detecting performance persistence if it exists.

 The first test of predictability is based on the correlation between performance measures of individual market advisory programs across pairs of marketing years.  The first step in the analysis is to rank each advisory service based on net price received. Then the services are sorted in descending order.  For example, the service with the highest net advisory price is ranked number 1, and the service with the lowest net advisory price is assigned a number equal to the total number of observations for that commodity in the given year.  Finally, the correlation coefficient is computed between the sorted performance measures for two adjacent marketing years.  A significant correlation indicates predictability in returns across years. 

Figure 2 presents a graphical illustration of the correlation across marketing years for corn, both in terms of advisory rank and percentage return above the 24-month market benchmark price.  Figure 3 shows the same relationships for soybeans, and Figure 4 for revenue.[17]  Estimated correlation coefficients and tests of significance are presented in Table 11.[18]  For corn, a significant and moderately positive correlation is found in the net advisory price and the percentage return above the 24-month benchmark between the 1995 and 1996 marketing years.  A positive correlation also is found between the rank of the services in corn between 1995 and 1996, but it is not statistically significant.  Nominally, just the opposite situation occurs for the 1996 and 1997 marketing years, where negative correlations are found for all three performance measures.  The net result is a small average correlation coefficient across the two pairs of years, about 0.10.  Hence, there does not appear to be consistent pricing performance across time in corn for individual advisory services.

Little evidence of predictability is found for soybeans.  All of the estimated correlation coefficients are positive, but only one is significantly different from zero (rank correlation, 1995 vs. 1996).  When averaged across the two pairs of marketing years, the correlation is only about 0.20.  Predictability results for revenue are similar to those found for corn and soybeans.  Overall, there does not appear to be evidence of persistence in the pricing performance of market advisory services.

 While the correlation analysis does not appear to find predictability in advisory service performance across all advisory services, it is possible that sub-groups of advisory services may exhibit predictability.  In particular, predictability may only be found at the extremes of performance.  That is, only top-performing services in one year may tend to perform well in the next year, or only poor-performing services may perform poorly in the next year.  To examine this form of predictability, market advisory programs are grouped according to performance in one marketing year, and their average performance in the next marketing year is evaluated.  Market advisory programs are grouped into quantiles of thirds and fourths.[19]

Quantile results for corn market advisory programs in the 1996 marketing year based on performance in 1995 are presented in Table 12.  When the programs are broken into three groups, the group in the middle third of advisory performance in 1995 performs the best in 1996 in terms of average price and average percentage return above the market benchmark.  The top third of advisory programs in 1995 has a slightly better average rank in 1996.  Similarly mixed results are found among the top and middle groups when the programs are broken into four groups.  While statistical significance is not assessed in this analysis, it appears that any real persistence in performance is found in the bottom group – i.e., market advisory programs that performed poorly in 1995 also perform poorly in 1996, both in terms of prices and rank.

The results of the 1996 and 1997 comparison for corn are presented in Table 13, and these results show a much more mixed picture.  When broken into three groups, advisory performance measures among the groups are virtually identical in 1997.  The quartile analysis contains a rather odd statistical anomaly, in which the first and third groups and the second and fourth groups in 1996 produce identical average prices in 1997, as well as similar ranks.  This does not argue for overall persistence in performance among the groups.

The soybean performance results for 1995 versus 1996, shown in Table 14, present a very similar picture to the 1995 versus 1996 corn results.  The main evidence of persistence in results is that services that do poorly in 1995 also show worse pricing performance in 1996.  The soybean results for 1996 versus 1997 presented in Table 15 also are similar to those for corn over the same years, in that little evidence of persistence is found.

Table 16 presents two-year average results of the persistence measure shown in Tables 12 through 15.  The advisory programs are grouped into quantiles each year (year t) and the average result in the next year (year t+1) is calculated.  The two-year average results indicate that any persistence in year-to-year performance is found only among the poorly performing advisory programs.  Based upon the results in Tables 12 through 15, it is obvious that the two-year results are mostly a function of the 1995 versus 1996 results.[20]

In general, the predictability results reported in this section provide little evidence that future advisory service pricing performance can be predicted from past performance.  The limited evidence in favor of predictability applies only to the poorest performing services.  This information may well be of use to farmers as they make selection decisions.  The similarity between the results for advisory services and mutual funds is striking. A number of studies find that mutual fund investment performance is not predictable in general, but that mutual funds ranked in the bottom tier in one year tend to remain in the bottom tier in the future (e.g. Brown, Goetzmann, Ibbotson and Ross, 1992; Carhart, 1997).  This has led researchers to search for an explanation of why investors continue to invest in mutual funds with predictably poor performance.  One explanation is myopic loss aversion on the part of mutual fund investors. (Odean, 1998). 

Finally, even though past net price performance does not appear to be useful in predicting future net price performance, this does not mean it is impossible to predict advisory service performance.  There may be other variables associated with performance that can be used for  prediction.  For example, Chevalier and Ellison (1999) study whether mutual fund performance is related to characteristics of fund managers that indicate ability, knowledge or effort, and find that managers who attended higher-SAT undergraduate institutions generate systematically higher returns.  Barber and Odean (2000) examine the trading records of individual stock investors and report that frequent trading substantially depresses investment returns.  Whether these types of factors can predict advisory service performance is an interesting question that awaits further research. 

Farmers view market advisory services as a significant source of market information and advice in their quest to manage price risks associated with grain marketing.  Given the high value that farmers place upon market advisory services, it is somewhat surprising that only two academic studies investigate the pricing performance of advisory services.  The lack of studies is most likely due to the difficulty in obtaining data on the stream of recommendations provided by services.

In 1994, the Agricultural Market Advisory Service (AgMAS) Project was initiated, with the goal of providing unbiased and rigorous evaluation of market advisory services for crop farmers.  Since its inception, the AgMAS Project has been collecting marketing recommendations for about 25 market advisory programs.  The AgMAS Project subscribes to all of the services that are followed, and as a result, "real-time" recommendations are obtained.  This prevents the data from being subject to survivorship and hindsight biases.

The purpose of this paper is to address two basic performance questions for corn and soybeans using the net price received reported by the AgMAS Project for the 1995, 1996 and 1997 marketing years.  The two basic questions are: 1) Do market advisory services, on average, outperform an appropriate market benchmark? and 2) Do market advisory services exhibit persistence in their performance from year-to-year?  At least 21 advisory services are included in the evaluations for each commodity and marketing year.  While the sample of advisory services is non-random, it is constructed to be generally representative of the majority of advisory services available to farmers.  The tests used to determine average performance of market advisory services and predictability of performance through time have been widely applied in the financial literature.

Tests of pricing performance relative to a market benchmark are based on the proportion of services exceeding the benchmark price and the average percentage difference between the net price of services and the benchmark price.  In statistical terms, the pricing performance test results provide little evidence that market advisory services consistently and significantly “beat the market” in corn.  There is substantial evidence that market advisory services consistently and significantly “beat the market” in soybeans.  When corn and soybean net advisory prices are combined into revenue per acre, some evidence also is found that market advisory services significantly outperform the market.  Tests results for revenue are the most sensitive to the type of test and benchmark considered.  Overall, the statistical results suggest that market advisory services have some ability to outperform broad market benchmarks.

It is debatable whether the performance of advisory services also is economically significant. Perhaps the best perspective on this question is gained by examining returns for corn and soybean revenue per acre.  For all three marketing years, returns averaged 0.74 percent above benchmark revenue, which translates into about $3 per acre.  While this level of return is probably best characterized as “small,” it also appears to be non-trivial, particularly in comparison to the cost of the services.  However, there are two important reasons to be cautious about concluding that advisory returns generate even a "small" level of economic significance: i) the results are based on a small sample of years, and ii) returns are concentrated in only one market, soybeans.

Tests of predictability are based on the year-to-year correlation of advisory service ranks, prices and percentage differences from the benchmark.  In general, the predictability results provide little evidence that advisory service pricing performance can be predicted from year-to-year.  The average correlation coefficient relating performance from one year to the next is about 0.10 to 0.20.  When services are grouped by performance quantile, some evidence of predictability is found for the poorest performing services, but not for top performing services.

In conclusion, the results of this study suggest that, on average, market advisory services exhibited some ability to "beat the market" for the 1995 through 1997 corn and soybean crops.  Possible explanations for this result include: i) a unique time period in corn and soybean markets, ii) inefficient commodity markets, iii) the skillfulness of advisory services or iv) a return to risk.  Determining which explanation is correct will be an important subject for future research as more data on market advisory performance becomes available. 

Anderson, D.R., D.J. Sweeney and T.A. Williams.  Statistics for Business and Economics, Sixth Edition.  West Publishing Company:  Minneapolis/St. Paul, 1996.

Barber, B.M. and T. Odean."  "Trading is Hazardous to Your Wealth: The Common Stock Investment Performance of Individual Investors."  Journal of Finance  55(2000):forthcoming. [http://www.cob.ohio-state.edu/htbin/htimage/~fin/journal]

Bodie, Z., A. Kane, and A.J. Marcus.  Investments.  Irwin: Homewood, IL, 1989.

Brorsen, B.W. and J. Townsend.  "Performance Persistence for Managed Futures." Proceedings of the 1998 NCR‑134 Conference on Applied Commodity Price Analysis, Forecasting, and Market Risk Management. Department of Agricultural Economics, Kansas State University, pp. 337-354.

Brown, S. J., W. Goetzmann, R.G.Ibbotson, and S.A.Ross. "Survivorship Bias in Performance Studies." Review of Financial Studies 5(1992):553-580.

Carhart, M.M.  "On Persistence in Mutual Fund Performance."  Journal of Finance 52(1997):57-82.

Chevalier, J. and G. Ellison.  "Are Some Mutual Fund Managers Better Than Others? Cross-Sectional Patterns in Behavior and Performance."  Journal of Finance 54(1999):875-899.

Clements, J.  "Debunking Some Mutual-Fund Myths."  The Wall Street Journal, March 16, 1999.

E. J. Elton, M. J. Gruber, and J. C. Rentzler. "Professionally Managed, Publicly Traded Commodity Funds." Journal of Business 60(1987):175-199.

Fama, E. "Efficient Capital Markets: A Review of Theory and Empirical Work." Journal of Finance 30(1970):1043-1053.

Gehrt, D. and Good.  "Evaluation of Market Advisory Services for Corn and Soybeans."  Journal of the American Society of Farm Managers and Rural Appraisers 57(1993):1-7.

Good, D.L., S.H. Irwin, T.E. Jackson, and G.K Price.  "1995 Pricing Performance of Market Advisory Services for Corn and Soybeans."  AgMAS Project Research Report 1997-01, March 1997. (http://www.aces.uiuc.edu/~agmas/evaluation_of_services/95cornbean/95report-abstract.shtml)

Good, D.L., T.E. Jackson, and S.H. Irwin. "Development of a Market Benchmark for AgMAS Performance Evaluations." AgMAS Project Research Report 1998-02, December 1998. (http://www.aces.uiuc.edu/~agmas/evaluation_of_services/98benchmark/98benchmark.shtml).

Grossman, S. and J. E. Stiglitz. "On the Impossibility of Informationally Efficient Markets." The American Economic Review 70(1980):393-408,.

Irwin, S.H., C.R. Zulauf, and B.L. Ward.  "The Predictability of Managed Futures Returns." Journal of Derivatives  2(1994): 20-27.

Jackson, T.E., S.H. Irwin, and D.L. Good.  "1996 Pricing Performance of Market Advisory Services for Corn and Soybeans."  AgMAS Project Research Report 1998-01, January 1998. (http://www.aces.uiuc.edu/~agmas/evaluation_of_services/96cornbean/96report-abstract.shtml)

Jackson, T.E., S.H. Irwin, and D.L. Good.  "1997 Pricing Performance of Market Advisory Services for Corn and Soybeans." AgMAS Project Research Report 1999-01, February 1999. (http://www.aces.uiuc.edu/~agmas/evaluation_of_services/97cornbean/97report-abstract.shtml)

Jaffe, J.F.  and J.M. Mahoney.  "The Performance of Investment Newsletters." Journal of Financial Economics 53(1999):289-307.

Kastens, T.L. and T.C. Schroeder.  "Efficiency Tests of July Kansas City Wheat Futures." Journal of Agricultural and Resource Economics  21(1996):187-198.

King, R.P., L.S. Lev and W.E. Nefstad.  "A Position Report for Farm-Level Marketing Management."  Review of Agricultural Economics 17(1995):205-212.

Lakonishok, J., A. Shleifer, and R.W. Vishny.  "The Structure and Performance of the Money Management Industry."  Brookings Papers:  Microeconomics  (1992):339-391.

Marcus, A.J.  “The Magellan Fund and market efficiency.”  Journal of Portfolio Management  16(1990):85-88.

Marten, J. "Farmers Want Market News, Not Advice."  Farm Journal Extra, June 1994.

Martines-Filho, J.G.  Pre-Harvest Marketing Strategies for Corn and Soybeans:  A Comparison of Optimal Hedging Models and Market Advisory Service Recommendations.  Unpublished Ph.D. dissertation, The Ohio State University, 1996.

Metrick, A.  "Performance Evaluation with Transactions Data:  The Stock Selection of Investment Newsletters."  Journal of Finance 54(1999):forthcoming. [http://www.cob.ohio-state.edu/htbin/htimage/~fin/journal]

Odean, T.  "Are Investors Reluctant to Realize Their Losses?"  Journal of Finance 53(1998):1775-1798.

Otte, J. "Marketing Matters -- How Well Do Market Advisors Deliver?"  Prairie Farmer, July 19, 1986.

Patrick, G.F. and S. Ullerich.  "Information Sources and Risk Attitudes of Large-Scale Farmers, Farm Managers, and Agricultural Bankers."  Agribusiness 12(1996):461-471.

Patrick, G.F., W.N. Musser, and D.T. Eckman.  "Forward Marketing Practices and Attitudes of Large-Scale Midwestern Grain Farmers."  Review of Agricultural Economics 20(1998):38-53.

Powers, L.  "How to Measure Your Pro's Performance."  Top Producer, April 1993, p. 17.

Ritter, J.R.  "The Long-Run Performance of Initial Public Offerings."  Journal of Finance 46(1991):3-27.

Schroeder, T.C., J.L. Parcell, T.L. Kastens, and K.C. Dhuyvetter.  "Perceptions of Marketing Strategies; Farmers vs. Extension Economists.  Journal of Agricultural and Resource Economics  23(1998):279-293.

Wisner, R.N., E.N. Blue and E.D. Baldwin.  “Preharvest Marketing Strategies Increase Net Returns for Corn and Soybean Growers.” Review of Agricultural Economics. 20(1998):288-307.

Zulauf, C.R. and S.H. Irwin.  “Market Efficiency and Marketing to Enhance Income of Crop Farmers.” Review of Agricultural Economics 20(1998):308-331.

Click here for Order Form