The Silicon Valley Bank State of the Industry Survey: How Reliable are its Responses?
Too often, desired conclusions skew analysis to produce self-fulfilling results. While keeping faulty data from contributing to sampling error is simply a matter of objectivity and basic arithmetic, more subtle are the subjective factors that yield non-random samples. In the case of the Silicon Valley survey, the psychological disposition of the respondents is a significant distorting factor.
At the February Oregon Wine Symposium in Portland, Silicon Valley Bank principal, Rob McMillan, gave a preview of the bank’s 2012 State of the Industry Report, projecting Oregon industry growth of between 7% and 11%.
In presenting his data, Rob noted that 80 of some 400 Oregon wineries had responded. This corresponds to a confidence level of 95% (a generally standard target for researchers) with a confidence interval of plus or minus 10%. Thus, with 80 of 400 wineries reporting, if the sample were random, there is a 95% certainty that the range of growth will be between -1% and 19% (plus or minus 10% against the reported average of 9%. This describes industry conditions ranging from stagnation to boomy growth, random indeed.
Adding to the uncertainty is the definition of sales growth. Is one speaking of unit growth or revenue growth? Unit growth may well be accompanied by a drop in revenue (“We lose a dollar on every sale but we make it up on the volume.”) Or, the data may just be GIGO (Garbage In / Garbage Out) as with the 2011 Oregon Winery and Vineyard Report covering 2010.
In that report, industry revenue was reported at $252M on 1.93M cases, which works out to $131 per case. In 2009, the respective figures were $202M and 1.66M cases or, $122 per case. In the environment of rampant discounting that existed in 2010, a growth in case value simply was not possible.
Expressed as a story problem, if in 2009, I sold 10 cases of $600 wine for total revenue of $6000, I would have had to sell $7500 of wine to achieve the 25% reported revenue growth of 2010. Average industry discounting of only 10% below 2009 pricing (not much $50 Oregon Pinot Noir was going at frontline pricing or at least going very fast), would have required a nearly 40% increase in case sales to achieve 25% dollar growth.
If one’s winery is failing, it is highly unlikely that receiving a business conditions survey in the mail is going to be very appealing. Who would want to be reminded, question by question, of how bad things are? Even those making ends meet don’t particularly want to see what others are presumably doing at the top of the scale. Rather, surveys such as Silicon Valley’s tend to be overly optimistic as, even anonymously, people want the world to know how well they are doing. Moreover, as the Silicon Valley survey gives the option of identifying one’s winery, there is even more incentive for the haves to trumpet their success.
Sociology also plays a role in sample error. Electoral surveys that are as statistically well-constructed as possible have two inherent sociological flaws. First, opinionated people are more likely to pick up a telemarketing call during dinner and thus don’t necessarily represent the deliberating voter. Second, as the preponderance of calls go to land lines, the results are slanted to an older, more traditional demographic.
Similarly, surveys of customer satisfaction are more likely to be completed by people either very happy or unhappy with their purchases. In the case of fashion or durable goods purchases such as automobiles, buyers may rationally cite the reasons for their decision but, in the end, people buy to satisfy the complex Gestalt of ego.
Survey responses tend to attract a non-random sample by their very nature: those who perceive something to gain by responding, those more desirous of having their opinion heard or those inclined to boast of their success or distinguish a particular prowess. These factors coupled with conclusion driven analysis and poor sampling methods yield only a superficial view, like the 50 mile setting on a car’s GPS, a level of detail anyone familiar with a map of the United States already knows but doesn’t provide sufficient direction for where one wants to go.
