This note was posted to the Bridge Laws Mailing List in April of 2002.
There’s been a great deal of discussion of late as to the role of Appeals Committees in the ACBL, with some suggesting that ACs be eliminated entirely.
In an attempt to ascertain whether committees are improving rulings overall I’ve undertaken an analysis of the Toronto casebook. PDF versions of NABC casebooks are available here.
I’d have liked to rely on the Expert Panel’s rating of the decisions. I found this was not practical, in large part because the Panel’s rating includes criteria other than the correctness of the decision. In particular the Panel tends to downgrade a committee decision when they judge that an "Appeal Without Merit Warning" was merited but not assessed. Instead I have reviewed each case individually, relying on the panel’s comments and ratings to help inform my own judgement. I have not judged that an AC improved a director’s ruling unless the panel gave the AC’s ruling a higher rating, and I have not judged that an AC worsened a director’s ruling unless the panel gave the director’s ruling a higher rating.
Rich Colker summarizes the appeals in the back of each casebook. He categorizes each decision as "Good" or "Poor". I didn’t find this sufficient for my purposes for two reasons. One is that some cases are too close to call -- even were I to disagree with such a decision I would not want to call it poor. Another is that his categorizations sometimes appear arbitrary to me.
My methods are not rigorous, and I’d appreciate any suggestions as to method or as to the categorization of particular cases. I’d also love to hear whether anyone has undertaken similar analysis for other tournaments, in the ACBL or elsewhere.
Here’s what I found:
There were 73 cases in Toronto. 36 were decided by Panels, 37 by ACs. Of the 37 the AC ruled as the director did 20 times. I examined the 17 cases where the AC ruled differently than the table director.
In three cases (30, 64, 65) the screening director improved the ruling made at the table, and the AC made the same ruling as the screening director.
In one case (37) the screening director worsened the ruling made at the table -- the committee improved both rulings.
In nine cases (33, 40, 41, 42, 49, 50, 60, 63, 68) the committee improved the ruling made at the table. On one of these, case 49, the committee made what I consider to be an illegal score adjustment. The committee never the less improved on the director’s ruling.
I judged four cases (5, 35, 48, and 62) as too close to call.
What surprised me was that I found not a single case where an AC clearly worsened the director ruling. Even in the four cases I judged too close to call the expert panel rated the AC ruling higher than they rated the table ruling. While there is surely room for improvement in the ACs, it seems that in terms of quality of rulings they are doing substantially more good than harm.