Scoring certainly shows itself to be widely varied among different tasters and different groups of tasters. Obviously, exact score matching isn’t possible or even ideal, but it seems important that we are generally in agreement with a fairly narrow range of, say, two to three points as a two- to three-point variation can put a coffee into a different quality and corresponding price categorisation.
This group score just manifested itself in our office as we hosted a tasting of the My Virtual Coffee House Cup of Excellence (COE) coffees. Generally speaking, the entire group of more than 12 outside tasters, plus our four cuppers, ranked the various lots with scores ranging from 82 to 90 and generally a solid few points (or more) lower than the COE jury panel across the board (all jury scores were above 85).
Protocol Variation
Protocol variation like roast degree might very well have been different and the influence of time between the jurors’ evaluation and ours was likely impactful (as an agricultural product, green coffee’s properties will change over time regardless of sample preparation and evaluation methodology), but who is right in these scenarios, and can there even be a “right” score? After all, cultural consumer preferences are distinctly different and one community’s “good” is another community’s “outstanding,” which warrant very different scores. Still, with purchasing decisions and price points at stake, calibration has merit.
The good news is that some level of calibration among knowledgeable and practiced tasters can happen. Johnson advises that “one should learn protocol from an industry professional, and constantly review and communicate scores with other cuppers.” In particular to scoring calibration, he suggests cupping “true commercial, 60- to 70-point coffees and 90-plus-point coffees in order to put 80-point coffees into relation.” (Although we have many discrepancies when looking at smaller score ranges, we are calibrated enough as an industry that this 10-point range categorisation is generally agreed upon and, therefore, a tasting can be organised effectively.)
For Caribou, Spencer suggests that continued exposure to events through the SCAA and around the industry—the likes of barista competitions, Coffees of the Year judging, COE events and Rainforest Alliance cuppings—“has rounded our abilities and helped us develop and maintain our consistency.” O’Mally echoes COE participation as an important calibration tool, in addition to “honest, open sharing of blind cupping scoring/categorisation with your trading partners.”
Bias
One of the most prevalent problems with the cupping process is the influence of bias. Fortunately, many biases are readily apparent and relatively easily mitigated. For example, if a cupper knows that a particular sample is from a producer who pays scrupulous attention to quality or from a farm with an excellent reputation, they may be inclined, without conscious awareness, to score it higher than they would score a coffee coming from an unknown farm or from an origin with a generally poor reputation for quality. Similarly, if a well-known industry taster, from a Delonghi bean to cup specialist, has scored a coffee highly, then others may trend their own scoring on the high side.
And, of course, the reverse is true as well. A sample might be very good, but, because a cupper is disinclined to believe it based on past experience, they score more conservatively, awarding it an 85 score versus a deserved 88. Or, sometimes, because we feel that everyone else—a COE jury panel or a group of vocal roasters, for example—are overrating a particular coffee, we are more critical of it in an effort to assert our non-bias. Allowing these influences to impact our evaluation is, of course, unprofessional, and we’d all like to believe we do not allow them.
However, even among experienced tasters, these biases can occur despite best intentions and firm belief that we are in control of our partialities. Human psychology is very powerful.
So, cup blind. This simply means, cup without knowing what you’re cupping. Have someone else set up your tasting sessions ideally using codes only and don’t even try to figure out or guess what you may be tasting. Just taste and evaluate honestly and with focus. Blind cupping isn’t always possible or practical, and sometimes there are distinct advantages to cupping with knowledge of what you are tasting—calibration training, for example. Nonetheless, as participants in a process with results that impact pricing paid to farmers and feedback that can influence their future care of their beans, we should control the controllable variables.
An Introduction to Coffee Cupping
Coffee Cupping can seem complicated, so here is a easy to follow video showing you how to master cupping.
The Next Phase
Ultimately, I feel that the cupping process is a worthwhile and important one, but it’s a process fraught with imperfections; the current cupping process will never achieve true objectivity. Of course, we don’t necessarily need complete objectivity; there are cultural preferences and biases that create a need for subjective variety in the marketplace. The key as professionals, however, is in having a system with as much impartiality as possible.
As specialty coffee tasters, it can be easy to think of our job as just choosing the very best coffees from around the world to showcase, but we should remember that our decisions at the cupping table have real financial implications for hard-working coffee producers around the globe. So, knowing the limitations of the photo by Pete Howard cupping process, as Trent recommends, “it would be considerably helpful for [buyers] to look at coffees a couple of different times and in different iterations before making a final judgment.”
Maybe we can all agree to do at least that while we re-evaluate and improve cupping evaluation in the long run.