The study carried out by Britisch psychologist and skeptic Richard Wiseman in February 2011 “proves” that people are unable to distinguish a good wine from plonk. Skeptic, wine amateur and psychologist myself, this grabbed my attention. I tried to find out what exactly prof. Wiseman had done.
In this article, I come to the conclusion that it might be true that people generally cannot distinguish between cheap and expensive wine, but that the Wiseman study is too weak to support that conclusion.
First, I give an overview of what the popular media reported about the study. Second, I describe how the experiment was actually set up and I describe some questions that we are left with after the study. Third, I present some evidence that people indeed cannot make the difference between cheap and expensive wine. And last but not least, I conclude that the Wiseman study does not advance science unless… but I’ll keep my hypothesis for the end.
1. General media coverage
Metro (Flemish Edition, Friday April 15th) states “Cheap wine is as tasty as expensive one”. “From 587 people from the public”, the newspaper continues, only 50% identified the wine correctly”. And further: “Only red wines like merlot and chardonnay (!) were tasted”.
The Guardian (online edition) states “apparently, some 578 lucky visitors [...] were given two glasses of wine, told that one of them was cheap and the other expensive, and asked which was which”.
BBCNews (online edition) says, “The blind test at the Edinburgh Science Festival saw 578 members of the public correctly identify the “cheap” or “expensive” wines only 50% of the time.”
The Daily Mail (Derbyshire, D, online edition) opens with “Cheap or nasty? Most Britons can’t tell the difference between plonk and fine wine”.
And continued: “The researchers approached 578 people in Edinburgh and asked them to sample a glass of either a cheap or an expensive wine. The tasters were shown two prices and asked to say which wine they were drinking. They correctly classified the white wines as cheap or expensive 53 per cent of the time, and the reds on 47 per cent of occasions. Overall they were right just 50 per cent of the time.”
The Daily mail also reveals details about the price level of the wines and the percentage correct per wine type. The story in the Daily Mail offers most details and is closest to what I think really happened. I tried to get confirmation of the experimental results by contacting prof. Wiseman several times, but he refuses to answer. So I’m forced to guess.
2. The experimental setup and open questions
Aside from the many factual errors (like different number of participants, the “red chardonnay” or the specification “French” champagne), there are two big misunderstandings that matter a lot.
The first misunderstanding is about the experimental setup itself. Depending on which source you consult, you are left with the idea that people were given two wines, a cheap one and an expensive one, and had to figure out which was which.
That was probably not the case. As the Daily Mail states, “We didn’t ask people to compare two wines because we wanted to mimic real-life conditions, said Professor Wiseman, of Hertfordshire University. When you have a meal you don’t decide whether a wine is good or not by comparing it with other wines – you drink it”.
A second misunderstanding has to do with the results. If you read the newspaper superficially, you would be left with the impression that half of the people cannot distinguish good from bad wine. So that would imply that half of the people *can*.
That also was not the case. If the Wiseman study proves anything, it is certainly not that there are two types of people: connaisseurs and ignorants. Several newspapers rightfully compare a 50% score on a discrimination task to “flipping a coin”, i.e. none of the participants scored better than luck. But we cannot conclude that from the data we have. All we know is that on average, people scored 50% at this discrimination task.
We don’t know whether there were people able to identify correctly 8 cheap wines from 8 more expensive wines. This is perfectly possible: all you need then, is enough people who think the cheap wine is expensive or vice versa to compensate. We are even not sure that people had to taste 8 wines or just 1. Also, we don’t know how many times a cheap wine was held for an expensive one or the reverse. The results were aggregated per wine type, so all we have is a report from Daily Mail with 8 wine types (four white and four red) with their percentage correct and sixteen prices ranging from £4.29 to £29.99.
3. Supporting evidence
Despite the confusion and the lack of quality data about the Wiseman study, the wine world reacted sharply. Jancis Robinson: “It is true that wines currently on sale for around £30 a bottle in British stores would typically be very young – too young – examples of wines that are meant to be aged for many years. A 2008 bordeaux classed growth, for example, would still be chock full of off-puttingly chewy tannins and youthful acidity. Only a professional would be likely to prefer this, for current drinking, to a soft, fruity £4.99 Chilean Cabernet Sauvignon.”
While I support this argument, I must say that Wiseman tried to choose comparable wines to avoid comparing a 2008 bordeaux classed growth to a soft and fruity chilean cab. For “claret”, the Daily Mail mentions a bottle of £3.49 and one of £15.99. That is probably a cheap, generic AOC bordeaux next to a saint emilion grand cru or a mediocre médoc. You can’t have high-quality bordeaux classed growth for sixteen pounds a bottle in a British supermarket. At the time of writing, there is one bottle of exactly that price available at Marks and Spencer and that is a cru bourgeois Château Devise d’Ardilley 2005 (Haut-Médoc). A wine that gets 3 stars in Decanter with the following tasting note: “rich, classy emphasis on cassis. Bold, bright and vivid. Crunchy fruit, firm tannins. Astonishingly undeveloped – promising. Drink 2016-23. (16.4 points)”.
Still, there is some evidence that people might prefer a cheap bordeaux over a more expensive one:
- There has been a study published in the Journal of Wine Economics in 2008 “Individuals who are unaware of the price do not derive more enjoyment from more expensive wine. In a sample of more than 6,000 blind tastings, we find that the correlation between price and overall rating is small and negative, suggesting that indiv iduals on average enjoy more expensive wines slightly less.”
- I carried out a blind tasting with a group of about 30 wine amateurs in which I chose 3 cheap (sub 5 euro) wines and asked four wine amateurs to find an expensive (about 25 euro) wine of the same type as my wine. If for each of the wines two-thirds of the tasters would prefer the expensive wine over the cheap one, I would lose. I won. My €3,55 Salice Salentino Cantine Due Palme 2001, notably a supermarket wine, was preferred over an expensive (€26,10) Salice Salentino Riserva 2001. Average scores were 13,1/20 for the cheap wine and 11,7/20 for the expensive one. While not set up in a scientific way, this study has one advantage over the Wiseman study: the selection process of the wines was much clearer. Indeed, both the “cheapo” (me) and the “luxury boys” (the four wine amateurs) had a good reason to pick the best wine they could for that price. There was no possible experimentor bias, as there probably was with the Wiseman study.
- There are numerous accounts of wine fraud by putting cheap wine into an expensive bottle. A case that springs to mind is the 2001 Schoonhoven fraud. Utterly cheap sauvignon from Chile ended up in a bottle of Château Schoonhoven, and was sold for about €100 in three-star restaurant Comme-chez-Soi. Even the knowledgeable sommelier William Wouters didn’t spot it. The fraud was dismantled after Justice found out that the number of bottles was far too large for the weight of the grapes bought to make the wine.
Here is what I believe:
- Non-wine experts generally like an other type of wine than experts. For them, it must be “not too sharp” and “good and soft” (I hear these words often in tastings with non-experts). Generally, this means cheaper, fruit-driven wines without too much tannin, or without the marked acidity a young quality wine can have. Exactly the point made by Jancis Robinson.
- You can fool a wine expert by showing an expensive bottle but pouring a cheap wine from it, or vice versa (pour Pétrus into a cheap bottle and ask the expert what he thinks – he might miss the subtlety of the wine completely). But in blind tastings, it’s impossible to fool all experts all of the time. Quality will shine through. I remember a blind panel tasting in 2006 where 38 experienced tasters judged 111 wines from two regions and two price categories (from about €3.5 to €39). The tasting was set up in four flights such that one flight consisted of wines from the same region and price category. The best wine, with a score of 93%, was also the most expensive and there was a positive correlation between price and perceived quality, even if the wines were young. Correlations between price and quality ranged from 0.29 to 0.58 depending on the origin and the price category of the wine. Strangely enough, the average correlation was the same for cheap versus expensive wines (0.43 and 0.44) but it was higher for wines from Southern France (0.53) than for Southern Italy (0.34). So in Southern France, you are more likely to get what you pay for than in the South of Italy. Wine-wise, that is.
- The lack of correlation between price and quality is partly due to demand/supply effects. For example, the demand for fine burgundy is ten times as high as the supply. Price goes up, quality goes down. The production cost of even the finest wines on earth rarely exceed €50. So anything you pay above that price (€2000 or more for a bottle of Pétrus 2009) is because demand is high, not necessarily because it is worth the price (Haut Brion 2009, another top bordeaux, gets comparable marks by wine experts but costs “only” €600 or so). This does not only hold within a region, but even more so between regions. I did a blind tasting with about 50 unexperienced wine lovers. They tasted 8 red wines from various European regions. I asked beforehand which of the regions they would like most. The general answer was, not surprisingly, “Bordeaux”. After the tasting, I asked which wine they actually liked the most. General feeling was that bottle number 4 was the best. Then I asked to say what region they thought it came from. Most thought it would have been bordeaux. When I revealed the label, it turned out we had tasted a €12 Languedoc wine. The bordeaux, also a €12 wine, came in last…
So in my opinion, it is quite possible to get the results prof. Wiseman achieved. What I do not support, is the claim that “cheap wine is as good as expensive wine”. There are far too many loose ends in his study to warrant such a far-fetching conclusion. Let’s discuss some open questions.
The experimental setup
A first cluster of open questions is about the experimental setup. How were people selected? How many wines did they taste? Were the bottles hidden with a brown bag, or were they decanted in a neutral caraffe? In what circumstances did this tasting take place? What glasses were used? Geoffrey Kelly said, according to Lucy Shaw of The Drinks Business “I found out that the wines were served in thimble-sized glasses – no wonder people were unable to pick out the fine wines from the cheap ones.”
What was the temperature of the wines? Who carried out the experiment and was this indeed carried out as a double-blind study, i.e. the participants don’t know it is an experiment, and the observer is unaware of the research hypothesis and the experimental conditions? The fact that prof. Wiseman is unwilling to share details about the experiment contributes to the opinion in the wine world that this is a very weak study, to say the least.
The experimental data
A second cluster of open questions is about the data. How were the data collected? Apparently, Wiseman spent no effort to note down individual scores (assuming that each person had to taste 8 and not only two wines). As stated by Gary Ernest Davis: “Imagine that half the people tested were experts and got it right every time, while the other half were dunces who got it wrong every time. On average, people would be right half the time”. Furthermore, how were “wine-loving” tasters determinations kept apart from the others to justify the claim that “even self-professed wine lovers failed to identify the wines correctly”? How was determined if a person was more experienced than others? There has been a study showing that non-expert are unaware of their incompetence but that more experienced people are aware of their lack of competence. This could mean that self-reporting on experience shows a negative correlation with actual performance. This is nicely illustrated by a certain mr. Jon from Staffs who reacts to the Daily Mail article online: “these so called experts dont even drink the wine.they slush it around there mouths and spit it out. im no expert, but even i know you have to swollow [sic] it!”. Where do you think Jon places himself with respect to wine knowledge, above or below the wine amateur whom he sees spitting out a wine?
Also with regard to this second cluster of open questions, the fact that prof. Wiseman is unwilling to share his experimental data weakens his study.
A third cluster of open questions handles the selection of the wines. As Jamie Goode points out, “Wines purchased from the producer for the same cost can end up at very different prices. A supermarket may pay a grower 1 Euro per bottle and then list the wine at £4.99 (a standard mark-up, given tax and logistical costs). They may then buy another wine for 1 Euro and list it at £7.99 or even £9.99, with a view to discounting it later.” Exactly! So having a precise mechanism to select “cheap” versus “expensive” wine is necessary. It is not sufficient to just run into a supermarket and buy eight pairs of bottles. At the very least, one should try to buy wines with a known production cost to rule out any pricing strategies supermarkets are known for. On top of that, there should be some protection from experimentor bias. If you know what the hypothesis is, it is not difficult to find wines that match this hypothesis. For example, I can go to the supermarket around the corner and pick out a very nice sauvignon for €4 and a bad sauvignon for €10. Furthermore, for non-experts, even £5 for a bottle of wine is fairly expensive, as illustrated by John W from Manchester, who says “I view a cheap bottle of wine will cost me £2.99, and an expensive will one cost £4.99″ as a reaction to the Daily Mail article. So why then compare “cheap” wines (on average £6.18) with “expensive” (on average £14.65). Why not compare £2.5 with £10? Why indeed not choose a larger difference (price tension = expensive divided by cheap, in this case on average 2.7 and ranging from 1.7 to 4.6)? And why not include expensive and cheap older wines into the sample, such that quality can express itself when the wine is ripe for consumption?
A strange correlation
A fourth open question is inspired by an observation I made when interpreting the scarce data from the Wiseman study. I discovered that the correlation between the price tension and the percentage correct is highly negative (-0.83). That means: for every pair of wines (cheap and expensive), if the difference in price is large, chances are that people actually think the cheap wine is the expensive one. When the price difference is small, people seem to have less trouble putting the right price label on the bottle. For example, the bordeaux pair has a price span of 4: the expensive partner of the pair costs 4 times as much (16 pounds) as the cheap one (4 pounds). With the pinot grigio pair, the price span is only 2.2 (cheap: £4.29; expensive £9.49) but here, 59% of tasters identified the wine correctly. This is strange and in fact contradicts the conclusion of prof. Wiseman, only not in a way that the wine world would like. It means that people can actually distinguish between cheap and expensive, be it that they label them wrongly. If price tension goes up, people have a higher probability than chance to declare the expensive wine cheap and vice versa. How does this come? It is difficult to interpret this result since we don’t have the raw data, but this finding is in line with the Goldstein 2008 results, i.e. that there is a negative correlation between price and perceived quality if laymen are quality judges. As Goldstein states, “Suppose we have two wines, A and B, and Wine A costs ten times more then Wine B in dollar terms.In terms of a 100-point scale (such as that used by Wine Spectator), [our model] predicts that non-experts will assign an overall rating that is four points lower for wine A, whereas experts will assign an overall rating that is seven points higher.
4. Bad science, after all
To conclude that “people cannot distinguish good wine from plonk” from the Wiseman study is totally unacceptable. On the one hand, there are serious methodological troubles with the experimental setup and with the experimental data, which should be available for closer inspection to the scientific community. The fact that professor Wiseman refuses to give them, is unprofessional and unscientific. On the other hand, there is strong experimental evidence for the far more interesting conclusion that laymen and experts judge a wine very differently. If the data as described by the Daily Mail are accurate reflections of the actual experimental data, these even support that conclusion. In that case, it can be assumed that the 578 person sample of professor Wiseman consisted of both laymen and wine experts, or at least amateurs.
Or is it?
How can it be possible that someone with a reputation as a skeptic like professor Wiseman sets up such a bad experiment, behaves unprofessionally with the data, and draws a totally invalid conclusion? I see two possible reasons. It was first stated by Jamie Goode: “Wiseman, a psychologist from the Hertfordshire University, is one of the seven-strong advisory group for the Edinburgh International Science Festival (www.sciencefestival.co.uk), and it seems that this ‘study’ was in essence a clever publicity stunt to boost the profile of the festival by generating column inches—one that worked very well. It was not designed as a proper scientific study, although this is how it has been reported”.
This would explain why Wiseman performed the experiment this way and reported about it with its invalid conclusion. But it would still be very unprofessional, both to the world of experimental psychology and to the wine world. To quote Goode again: “And it would be a shame if, through a publicity stunt dressed up as a piece of scientific research, people were put off exploring wine because they were led to believe that differences in wine quality are actually illusory”. Very true.
So I suspect that there is another reason for the publication of this bad experiment with its invalid conclusion. I guess the whole media storm is one of professor Wiseman’s experiments, a meta-experiment you could call it, to see how people react when under attack. I hope I am right, otherwise the skeptic world has just lost an important proponent to me.
BBCNews. Cheap wine ‘good as pricier bottles’ – blind taste test. Online at http://www.bbc.co.uk/news/uk-13072745.
Caldwell, E., Vintage Wine or Cheap Plonk? Science Festival reveals experiment results. Online at http://www.sciencefestival.co.uk/news/everyday/vintage-wine-or-cheap-plonk-science-festival-reveals-experiment-results.
Davis, G. (2011). Tasting Wine: a Coin Toss? Online at http://www.blog.republicofmath.com/archives/4720
Derbyshire, D. (2011). Britons can’t tell the difference between a fine wine and plonk. Online at http://www.dailymail.co.uk/sciencetech/article-1376686/Britons-tell-difference-fine-wine-plonk.html
Devriendt, E., Doomen, P., and Van Campenhout, S. (2006). Very Interesting Bottles uit Zuid-Frankrijk en Zuid-Italië. Ken Wijn-magazine nr 6, juni 2006.
Doomen, P. (2005). Proefverslag proeverij 4: Top of Flop. Available with the author.
Goldstein, R, Almenberg, J et al. (2008). Do more expensive wines taste better? Evidence from a Large Sample of Blind Tastings, Journal of Wine Economics, vol. 3, n. 1, p 1-9.
Goode, J. The Wiseman Study: cheap versus expensive wine. Online at http://www.wineanorak.com/wineblog/wine-science/the-wiseman-%E2%80%98study%E2%80%99-%E2%80%93-cheap-versus-expensive-wine
Guardian: Cheap wine, don’t be a plonker. Online at http://www.guardian.co.uk/lifeandstyle/wordofmouth/2011/apr/15/cheap-wine-dont-be-plonker
Robinson, J. (2011). What is wine value? Online at http://www.jancisrobinson.com/articles/a201104262.html
Shaw, L. (2011). Industry attacks psychologist for flawed taste test. The Drinks Business. Online at http://www.thedrinksbusiness.com/index.php?option=com_content&task=view&id=12634&Itemid=0
Supermarketwine (2011). If you can’t tell the difference between cheap and expensive wine, why pay more? Online at http://www.supermarketwine.com/blog/if-you-can-t-tell-the-difference-between-cheap-and-expensive-wine-why-pay-more
Thomas, L. (2011). Are wine buffs for real. TropicPost. Online at http://www.tropicpost.com/are-wine-buffs-for-real/