CGC Grading Contest Score History and Rankings 1 1

[Superman2006]

I was curious to see (a) how my grading contest scores have changed from contest to contest and (b) how my average grading contest skills stack up against others.

I mocked up the following Summary template that could be used to meet both of those objectives.

Since not all boardies will have competed in all 6 contests to date, the "Total Contests" column shows how many contests each boardie competed in, to give some sense as to the credibility of a boardie's average score. 

The "Contest Average" is calculated as the Total of the "Scores by Contest" / Total of the "Rounds Completed by Contest" * Total rounds per contest , e.g. in my case 96 / 20 * 4 = 19.20

If for whatever reason I had only partook in the first 2 rounds in contest #6, and my score for those 2 rounds was 10, then my total would have been 89 (instead of 96), and my rounds completed would have been 18 (instead of 20), giving me a "Contest Average" of 89 / 18 * 4 = 19.78. In other words, the approach I have proposed for calculating "Contest Average" works even if someone didn't complete all of the rounds in a given contest.

I figured I'd check to see if anyone else was interested in seeing this type of a grading score summary / ranking before trying to pull the data together into Excel and populating the summary below any further.

If someone was interested in becoming a grader for CGC and they had a good grading contest average score and decent number of total contests (or improved performance over time), that might not be a bad thing to mention in their cover letter / resume.

	Contest	Total		Scores by Contest								Rounds Completed by Contest
Boardie	Average	Contests		#1	#2	#3	#4	#5	#6	Total		#1	#2	#3	#4	#5	#6	Total
Superman2006	19.20	5.0		0	18	20	21	20	17	96		0	4	4	4	4	4	20

[Superman2006]

I just saw that zzutak posted another type of ranking across multiple contests on Monday; somehow I missed that post until now. It looks like we both had the same idea in mind for my item (b) above, albeit with slightly different approaches to coming up with a ranking.

[zzutak]

Here are a couple of my opinions on what constitutes a "good" grader.

Point #1:
My experience with CGC dates back to the early 2000s, when three different staff members independently assigned a numerical grade to each book.  Any collector could easily learn what those three grades were with a simple (and very short) phone call to CGC.  I made several of those calls to assess whether a CGC certified/encapsulated book I was considering purchasing was, for example, "a weak 8.0" (7.5, 8.0, 8.0), "a solid 8.0" (8.0, 8.0, 8.0), or "a strong 8.0" (8.0, 8.0, 8.5).  My experience with CGC convinced me that CGC's "system" (the precision of its condition grading rubric, and the skill of the individuals who are tasked with properly interpreting and applying that rubric) is only reproducible to + or - one grade increment, at best.  Hence, any player with a total score of 20 or less in any single CGC Contest is almost certainly an outstanding grader -- as good or better than most of the individuals CGC currently has on staff.

Point #2:
Back in the mid-2000s, Ed Jaster (who had recently joined Heritage Auctions) spent a few days at my home in San Luis Obispo.  He told me the "trick" to having your books certified by CGC was to max out your invoices rather than submit only a few books.  "Sometimes you'll be higher than CGC, and sometimes you'll be lower; but if you're a good grader, your average grade should come pretty close to CGC's."

CGC's proprietary (and still unpublished) grading rubric necessarily allows for judgment, and different graders may interpret/apply a given standard differently.  Blemishes/defects can be overlooked or discounted for any number of reasons.  The bottom line is that a given book may not always receive the same CGC grade if it's submitted several times (although the various grades assigned will almost certainly be within an increment or two of each other).  This fact (not opinion) will be obvious to anyone who's done an initial prescreen and then had the rejected books accepted on a later, identical, prescreen.  This is not meant to be a slam against CGC; I know of no collector who can/will assign the exact same grade to every one of his/her books every time he/she looks at 'em.

So, just for kicks, recalculate your total contest score using "signed deltas": -2 for your guess being 2 grade increments too low, -1 for your guess being 1 grade increment too low, 0 for a bullseye, +1 for your guess being 1 grade increment too high, +2 for your guess being 2 grade increments too high, and so on.  What's your cumulative 20-book tally now?  Did the errors to the low side and high side more or less balance out?  Or is your tally way negative or way positive (in which case you're consistently being too strict or too lenient with your condition assessments)?

Chances are your score using signed deltas will be much smaller than your score using absolute/unsigned deltas.  If that's the case, your errors are balancing out (which is exactly what Ed predicted would happen for a decent grader).  Here's an example using my own Contest #3, Round 1+2+3+4 guesses (20 books):

• 5 Bullseyes = 0
• 3 times my guess was one grade increment too low = -3
• 7 times my guess was one grade increment too high = +7
• 3 times my guess was two grade increments too low = -6
• 2 times my guess was two grade increments too high = +4

My total signed delta score would be 0 - 3 + 7 - 6 + 4 = +2 for the 20-book challenge.  That's an average error of one-tenth of one grade increment.        Compare that to my total absolute/unsigned delta score (my actual contest score) of 20, or an average error of one grade increment per book.  This is exactly what Ed Jaster predicted would happen for a person whose grading standards are essentially in line with CGC's.

Edited May 10, 2023 by zzutak

[grendelbo]

On 5/10/2023 at 10:37 AM, zzutak said:

Here are a couple of my opinions on what constitutes a "good" grader.

Point #1:
My experience with CGC dates back to the early 2000s, when three different staff members independently assigned a numerical grade to each book.  Any collector could easily learn what those three grades were with a simple (and very short) phone call to CGC.  I made several of those calls to assess whether a CGC certified/encapsulated book I was considering purchasing was, for example, "a weak 8.0" (7.5, 8.0, 8.0), "a solid 8.0" (8.0, 8.0, 8.0), or "a strong 8.0" (8.0, 8.0, 8.5).  My experience with CGC convinced me that CGC's "system" (the precision of its condition grading rubric, and the skill of the individuals who are tasked with properly interpreting and applying that rubric) is only reproducible to + or - one grade increment, at best.  Hence, any player with a total score of 20 or less in any single CGC Contest is almost certainly an outstanding grader -- as good or better than most of the individuals CGC currently has on staff.

Point #2:
Back in the mid-2000s, Ed Jaster (who had recently joined Heritage Auctions) spent a few days at my home in San Luis Obispo.  He told me the "trick" to having your books certified by CGC was to max out your invoices rather than submit only a few books.  "Sometimes you'll be higher than CGC, and sometimes you'll be lower; but if you're a good grader, your average grade should come pretty close to CGC's."

CGC's proprietary (and still unpublished) grading rubric necessarily allows for judgment, and different graders may interpret/apply a given standard differently.  Blemishes/defects can be overlooked or discounted for any number of reasons.  The bottom line is that a given book may not always receive the same CGC grade if it's submitted several times (although the various grades assigned will almost certainly be within an increment or two of each other).  This fact (not opinion) will be obvious to anyone who's done an initial prescreen and then had the rejected books accepted on a later, identical, prescreen.  This is not meant to be a slam against CGC; I know of no collector who can/will assign the exact same grade to every one of his/her books every time he/she looks at 'em.

So, just for kicks, recalculate your total contest score using "signed deltas": -2 for your guess being 2 grade increments too low, -1 for your guess being 1 grade increment too low, 0 for a bullseye, +1 for your guess being 1 grade increment too high, +2 for your guess being 2 grade increments too high, and so on.  What's your cumulative 20-book tally now?  Did the errors to the low side and high side more or less balance out?  Or is your tally way negative or way positive (in which case you're consistently being too strict or too lenient with your condition assessments)?

Chances are your score using signed deltas will be much smaller than your score using absolute/unsigned deltas.  If that's the case, your errors are balancing out (which is exactly what Ed predicted would happen for a decent grader).  Here's an example using my own Contest #3, Round 1+2+3+4 guesses (20 books):

• 5 Bullseyes = 0
• 3 times my guess was one grade increment too low = -3
• 7 times my guess was one grade increment too high = +7
• 3 times my guess was two grade increments too low = -6
• 2 times my guess was two grade increments too high = +4

My total signed delta score would be 0 - 3 + 7 - 6 + 4 = +2 for the 20-book challenge.  That's an average error of one-tenth of one grade increment.        Compare that to my total absolute/unsigned delta score (my actual contest score) of 20, or an average error of one grade increment per book.  This is exactly what Ed Jaster predicted would happen for a person whose grading standards are essentially in line with CGC's.

        

Plus we're going off of scans where it's really hard to get the "feel" of a book.

[Superman2006]

On 5/10/2023 at 12:37 PM, zzutak said:

Here are a couple of my opinions on what constitutes a "good" grader.

Point #1:
My experience with CGC dates back to the early 2000s, when three different staff members independently assigned a numerical grade to each book.  Any collector could easily learn what those three grades were with a simple (and very short) phone call to CGC.  I made several of those calls to assess whether a CGC certified/encapsulated book I was considering purchasing was, for example, "a weak 8.0" (7.5, 8.0, 8.0), "a solid 8.0" (8.0, 8.0, 8.0), or "a strong 8.0" (8.0, 8.0, 8.5).  My experience with CGC convinced me that CGC's "system" (the precision of its condition grading rubric, and the skill of the individuals who are tasked with properly interpreting and applying that rubric) is only reproducible to + or - one grade increment, at best.  Hence, any player with a total score of 20 or less in any single CGC Contest is almost certainly an outstanding grader -- as good or better than most of the individuals CGC currently has on staff.

Point #2:
Back in the mid-2000s, Ed Jaster (who had recently joined Heritage Auctions) spent a few days at my home in San Luis Obispo.  He told me the "trick" to having your books certified by CGC was to max out your invoices rather than submit only a few books.  "Sometimes you'll be higher than CGC, and sometimes you'll be lower; but if you're a good grader, your average grade should come pretty close to CGC's."

CGC's proprietary (and still unpublished) grading rubric necessarily allows for judgment, and different graders may interpret/apply a given standard differently.  Blemishes/defects can be overlooked or discounted for any number of reasons.  The bottom line is that a given book may not always receive the same CGC grade if it's submitted several times (although the various grades assigned will almost certainly be within an increment or two of each other).  This fact (not opinion) will be obvious to anyone who's done an initial prescreen and then had the rejected books accepted on a later, identical, prescreen.  This is not meant to be a slam against CGC; I know of no collector who can/will assign the exact same grade to every one of his/her books every time he/she looks at 'em.

So, just for kicks, recalculate your total contest score using "signed deltas": -2 for your guess being 2 grade increments too low, -1 for your guess being 1 grade increment too low, 0 for a bullseye, +1 for your guess being 1 grade increment too high, +2 for your guess being 2 grade increments too high, and so on.  What's your cumulative 20-book tally now?  Did the errors to the low side and high side more or less balance out?  Or is your tally way negative or way positive (in which case you're consistently being too strict or too lenient with your condition assessments)?

Chances are your score using signed deltas will be much smaller than your score using absolute/unsigned deltas.  If that's the case, your errors are balancing out (which is exactly what Ed predicted would happen for a decent grader).  Here's an example using my own Contest #3, Round 1+2+3+4 guesses (20 books):

• 5 Bullseyes = 0
• 3 times my guess was one grade increment too low = -3
• 7 times my guess was one grade increment too high = +7
• 3 times my guess was two grade increments too low = -6
• 2 times my guess was two grade increments too high = +4

My total signed delta score would be 0 - 3 + 7 - 6 + 4 = +2 for the 20-book challenge.  That's an average error of one-tenth of one grade increment.        Compare that to my total absolute/unsigned delta score (my actual contest score) of 20, or an average error of one grade increment per book.  This is exactly what Ed Jaster predicted would happen for a person whose grading standards are essentially in line with CGC's.

I think it's pretty safe to say that any player with a total score of 20 or less in any single CGC contest is a good grader. Although with "just" 20 books, I think it is possible that the stars just aligned for them on a single contest, but if they repeat that performance (or don't deviate too far from it) over multiple contests, then I think it's pretty safe to say they're an outstanding grader.

Regarding your 2nd point, a low total signed delta score on its own doesn't necessarily mean that someone is a good grader, as even a terrible grader could end up with a total signed delta score that is close to zero (I'm not suggesting that you implied otherwise). I agree that the total signed delta calculation is a good exercise for anyone to perform to see if they are consistently over or under-grading a comic compared to CGC, and something that could help them improve their grading skills in the future.

e.g. A not so great grader with a total score of 56 could still end up with a total signed delta score of zero if they threw darts to come up with their grades similar to the following;

• 0 Bullseyes = 0
• 1 times a guess was 1 grade increment too low = 0
• 3 times a guess was 1 grade increment too high = +3
• 3 times a guess was 2 grade increments too low = -6
• 2 times a guess was 2 grade increments too high = +4
• 3 times a guess was 3 grade increments too low = -9
• 1 times a guess was 3 grade increments too high = +3
• 2 times a guess was 4 grade increments too low = -8
• 3 times a guess was 4 grade increments too high = +12
• 1 times a guess was 5 grade increments too low = -5
• 0 times a guess was 5 grade increments too high = 0
• 0 times a guess was 6 grade increments too low = 0
• 1 times a guess was 6 grade increments too high = +6
• Total signed delta score = 0