A friend of mine just lent me a copy of the November 2005 issue of Consumer Reports. There is a review of summer, all-season, and winter tires in it. The results were... interesting, and there were some surprises.
One of the things I have discovered over the years is that the Consumers Union (the test lab that publishes Consumer Reports) strives for consistency in their measurements. This has both good and bad sides. On the good side, it helps ensure the integrity of their results. They are very good at taking data, processing it, and then presenting it. The down side is that the data they choose does not always give the full picture. I will frequently find that their ratings on various things match neither mine nor those of other sources, (including other publications.
As an example, if I were to just look at the quantitative data from our first Real World Tire Test, I would say that the Dunlop Graspic DS-1 was the winner. Braking, Acceleration, and ice performance where above the others. Qualitatively, the tire left some things to be desired, and as such did not rate that well overall.
In most cases, Consumer Reports is very good at providing some verbiage around their reviews so that you can better understand how they chose to rate things. From reading this, I usually get a feeling for what they are looking for.
My disappointment with their review of snow tires is not just because they rated one of the perennial top performers in our Real World Snow Tire Tests poorly, but because the verbiage to back up their ratings was lacking. There was little to explain how they tested and what they were looking for.
This brings up an interesting point that a tire representative shared with me. When considering winter tires, there are different aspects that different groups of people look at. Personally, I tend to want something that is going to give me an edge in running winter rallies. To that end, I want great performance at 40 - 60 mph on 1 - 3" of snow, moderate performance on ice, and acceptable performance on dry. Someone who lives in a city and only occasionally has to venture out on snow covered streets may put the emphasis on dry performance, but want a bit more safety when presented with the occasional snow storm. (Even here, I am reluctant to suggest an all-season tire.) A farmer who lives in the snow belt may put the emphasis on something that will go through 8" of snow even if it is slowly.
When we, the testers, write up the Real World Snow Tire Tests, we try to present both our comments about the tires and our biases. I have been tempted in the past to say that the tests are "unbiased." As long as there is a qualitative component of them, they can't be unbiased. Everyone brings a bias to the table. What we can say is that we are unbiased toward brands but biased towards what we want out of the tire. We can also try to explain what our biases are and how they effected the ratings. This is where I found the Consumer Reports article lacking.
Conclusion (or "Executive Summary")
The Consumer Reports article is a good read, but it should be taken with a grain of salt. One must understand that it is from a clinical view. It presents quantitative data. It does make some very good points about the need for snow tires and the performance differences between winter tires, all-season tires, and summer tires. It is lacking in a fair amount of detail that would help the reader understand the ratings.