• When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network.

Set Scores
3 3

47 posts in this topic

On 9/17/2021 at 2:37 PM, Figment said:

Another idea would be to simply say that a set must be at least x% complete before being considered for registry award ranking.  I'm not sure what value is appropriate for "x", but something that would generally eliminate sets of "one or two".

That's actually quite elegant. It should be easy to implement, and allows Brian to kick the can down the road on a point system revision that, to be honest, is probably both a more complicated problem and a more complicated solution.

Link to comment
Share on other sites

I really do appreciate the admins considering this issue and kicking around ideas on what can be done.

No one here knows how the backend system is coded up, so most of the suggestions we might think up could very well be impossible to implement (at least with a reasonable budget & turnaround time), but so far I love the idea of minimum % completion required to be considered for registry award ranking (as this technically changes nothing in the backend databases, but rather is mostly a front end check & styling - you can have all the sets that hit the % threshold appear at the top, and the rest that don't appear below them unnumbered/unranked and possibly visually different from the competitive ones above, perhaps slightly greyed out). As a web developer I can see this being the most feasible and the least amount of work needed on the site code itself / increasing database calls.

As for other suggestions, I still maintain that pedigree books should offer some sort of point bonus (even small) over the equivalent Universal book. CGC spends lots of effort promoting and listing all the comic pedigrees on their website, so it makes sense to treat them like they do signature series books on the registry.

I also still feel allowing NG books to slot into the registry is a good idea, if only to allow those without deep wallets to feel they have a chance to complete a set % (somebody who manages to get an incomplete coverless Action Comics #1 should have that count for something in a set, even if it is worth zero points as it would allow us to raise the completion %). I stress this as a way to be more inclusive towards the many people entering this growing hobby - no sense is making people feel they are excluded from ever having a chance at building a full set before they even start collecting :)  

Edited by Sauce Dog
Link to comment
Share on other sites

On 9/21/2021 at 5:26 PM, Sauce Dog said:

No one here knows how the backend system is coded up, so most of the suggestions we might think up could very well be impossible to implement (at least with a reasonable budget & turnaround time)

 

^^^ This.

 

I brought up this topic at our bi weekly registry meeting today.

I sent @Qalyar's post detailing how some of the keys are affecting set scoring overall to my boss, he will review it and get back to me.

The most popular option among my colleagues is removing the ridiculously scored issues from competition.  I do understand this may upset some people, However, I think this will probably be the most cost effective and fair option of them all.  So what would be a fair cutoff?  5000?  10000? We may have to do a deep dive into the database to determine average total set scores, and compare that to the key/grail scores.  (I am not a DBA, so I would have to work with my boss and do a lot of research to even begin to code this.)

I really want to come up with a solution that satisfies the customers, and at the same time be reasonable to the higher ups.  I spend more time than I should on scoring issues, whether its adjusting the score for one issue/series that happens to pop, or finding a score for obscure sets. 

 

 

More to come....

 

Thanks!

-Brian

 

 

 

Link to comment
Share on other sites

On 9/23/2021 at 4:41 PM, wytshus said:

I brought up this topic at our bi weekly registry meeting today.

I sent @Qalyar's post detailing how some of the keys are affecting set scoring overall to my boss, he will review it and get back to me.

 

 

 

Thanks so much for your efforts in tackling this beast of a project, I work with similar things and appreciate the kind of long term headache they can be for others :D

I do agree that capping of the scores does sound like the easiest solution, and while I am not privy to all the numbers, I have the feeling that capping it out at 10,000 might suffice (as you can always go down more in the future but it might be harder to re-adjust to a higher number later if the lower didn't work out. But like you said, this will require looking at the grails/averages for a good number).

The registry first and foremost is a place to have fun building a set, not necessarily winning outright, so it should incentivize people to try to build new complete collection they can show off (that is the big thing, showing off a fun looking set). Lots of people I talk to are happy with trying to get into the top 25 or whatever of a set (since winning would require grades out of their budgets) but most importantly want to aim for building a 100% set in whatever way they can get. Giving a completion % point bonus would go a long way, but for the short term maybe simply giving those who get 100% (regardless of if they are a top score or not) a small trophy/star/100 icon or something else visual on their set to make it stand out (we are simple folks, a shiny blue ribbon could keep any of us happy even if we don't win) :D  A graphical icon would mean no database work (only checking if completion = 100%) but make it look spiffy 

 

Link to comment
Share on other sites

Hey Everybody,

My boss got back to me, and he has some questions.  I would appreciate input from you all before I respond.

-------------------------------------------------

 

Which method have our rivals chosen? E.g. PSA, PCGS, BGS (I know PCGS uses the GPA of grades in a set as a factor)


How a scoring change will affect our current sets – will it upset a lot of people? Once we have a few ideas whittled down, I can see in the DB what %age of sets this will change. 

 

Any thoughts about collection scoring vs set scoring. I see a trend in coin collecting towards trophy-hunting

 

--------------------------------------------------------

Anyone here collect coins as well?  I'm really interested in your thoughts about scoring between comics and coins.  I realize they are completely different animals, but it looks like the coin scoring is a bit more complicated, but it seems to be more equitable.

 

Thanks!

 

Link to comment
Share on other sites

On 9/28/2021 at 8:49 AM, wytshus said:

Which method have our rivals chosen? E.g. PSA, PCGS, BGS (I know PCGS uses the GPA of grades in a set as a factor)

  • BCS (cards): Beckett's card registry does not incorporate FMV at all. Scores are based exclusively on the grade and subgrades of each card, with a small point bonus for 10-graded cards that have a full set of 10 subgrades. This means that the maximum score for a set is a little higher than 10 times the number of cards in the set. A card worth thousands of dollars and a card worth a couple bucks, in the same grade, receive the same number of points.
  • NGC (coins): NGC's scoring works more or less like CGC's does, which should surprise no one since they're another CCG company. Like with CGC's registry, points are attributed based on an estimate of absolute FMV, and so big rarities amidst a series of low-dollar items have an oversized impact on the set scoring. That's somewhat less obvious for coins than for comics, because there are relatively few short coin runs that have one really rare coin and a bunch of drek, as it were.
  • PCGS (coins): PCGS incorporates FMV indirectly. When a set is created, each coin in the set is assigned a weight based on its FMV compared to other cards in the set. For most sets, this is on a scale of 1-5. For sets where there is an exceptionally large range in values between set members, the weighting can be on the scale of 1-10 instead. For example, for the Lincoln cents circulation strikes, common coins are a 1, the famously difficult 1909-S-VDB is a 7, and the 1914-D (unquestionably the hardest coin in the set) is an 8. Coins in a set are given a number of points equal to their rating times a fractional quality factor, between 0 and 1, based on their grade. This is somewhat more complicated than CGC grading because the numismatic grading scales are sometimes weird. Each coin can also earn 1 bonus point (for being tied for top of population) or 2 bonus points for being the unique top of pop. Some sets have additional bonus points available for non-numeric quality: for example, some copper coin sets give a 1 point bonus to red-brown coins and a 2 point bonus to red coins. This is roughly the equivalent of giving bonus points for off-white or white page quality.
  • PSA (cards): PSA is very similar to PCGS, which makes sense because they are both Collectors Universe subsidiaries. The quality factor is a lot easier to manage here because card grading works like comic grading and not the crazytown world of numismatic grades. Each card in the registry is worth it's weight value time (it's grade divided by 10). So, in a Magic: the Gathering set, a Beta Ancestral Recall (weighting factor 9) in 8.5 is worth (9 * 8.5 / 10 = 7.65 points). PSA also includes the top-of-pop bonuses in the same manner as PCGS.
  • Other collectibles grading companies, including CBCS (comics), ICG (coins), PGX (comics), and WATA (video games) do not appear to have registry systems at this time.

Honestly, my opinion is that the Collectors Universe (PCGS/PSA) system is probably the best of them. CGC/NGC's benchmarking of points to absolute market value means that many sets are fundamentally referendums on owning the handful of key chase items, rather than actually collecting the "set". BCS's absolutely flat point system swings the pendulum too far the other way.

I don't know what options the CGC registry has for significant revision. A lot depends on how the database was initially structured. As a professional data analyst, I'd be happy to consult! lol

Link to comment
Share on other sites

On 9/28/2021 at 12:00 PM, Qalyar said:

I don't know what options the CGC registry has for significant revision. A lot depends on how the database was initially structured. As a professional data analyst, I'd be happy to consult! lol

Dude, that is awesome.  I will definitely talk to my boss about it!  

Thank you!

Link to comment
Share on other sites

On 9/28/2021 at 9:49 AM, wytshus said:

 


How a scoring change will affect our current sets – will it upset a lot of people? 

 


I personally feel that those who are content with only a single high grade key from a run (and no other issues) are less concerned about the registry and just doing it to own the key issue - so while their epic book might be struck down in points it won't matter much to them (if they truly want to compete they can go and buy more than one single book for a set to actually try - it should never be a one-and-done purchase to win a set with a single book), whereas the collectors who are trying to build complete collections (not even necessarily trying to be the top scoring set due to budget) are the ones who are big fans of those characters, and I suspect the majority, and would love a chance to feel they can be competitive. 
 

Link to comment
Share on other sites

I still like the idea of a bonus based on completion percent but understand it may not be feasible depending on the design of the database.  It would certainly be easy to implement from a reporting perspective as all the required information, score and %complete, is already present.

As to upsetting people, well... change of any kind usually upsets someone, but I do think that if the change details and reason for change are clearly and accurately communicated it will go a long way in soothing any ruffled feathers.  Similarly, the message that registry awards will only be given for sets that are x% complete (in order to encourage set building) should be relatively well received and easy to understand because one issue does not a set make.

My feeling is that "x" should be 10%.  In that way, only a single book is required for sets of size 1-10, two books is required for sets of size 11-20, three books is required for sets of size 21-30, and so on.

Regardless of what change is made, IF a change is to be made then it should be publicized WELL in advance of implementation.  There should be announcements and banners and notices in the control panel, etc, etc, etc.  That way the initial "sting" or knee-jerk reaction for those that don't like it will have a chance to soothe over time before it becomes reality.  It will also give those folks a chance to "catch up" if they need to do so before they lose rankings.

Link to comment
Share on other sites

I agree that it feels a little incongruous that a few key comics can account for 50% or more of the total points value for a particular registry set. Nevertheless it is important the registry should, as the website states, reflect the scarcity, desirability and certified grade of each comic. Therefore I would not support the idea that there should be a scoring cap or particular comics should be completely excluded from the registry because of their extreme high value.

As a long term collector and a collector of runs I am signed up to the notion that there should be some recognition of completion rates for particular sets. Some excellent suggestions have been put forward and there seems to be consensus that any change should be well publicised and simple to administer. If this were to be developed I would suggest the maximum completion bonus should be restricted to a percentage of the total value of each set, perhaps around 25%.

One issue that doesn't appear to have been raised in this regard is the potential distorting effect of the same comic in different sets - and I confess that I do include the same comic in different sets myself. The existing rules are that if the same comic is entered in more than one set, its points are only counted once for the purpose of calculating Total Registry Points. The proposition of a completion percentage for different sets would therefore appear to create a loophole whereby the same comic in different sets would be credited as part of the completion bonus for each set, and therefore be aggregated into Total Registry Score. A radical, but no doubt deeply unpopular, solution would be to restrict each comic to being allowed a slot in one set only. However a concomitant benefit of this approach would be to prevent owners from using the same few comics to monopolise the top ranks in different, but similar, sets. To give an obvious example there are numerous sets which include Amazing Fantasy #15.

Interested to know if fellow collectors have any thoughts on that suggestion?

I also agree it is timely to recognise the desirability and generally higher prices commanded by comics from CGC recognised Pedigrees. Any change to the scoring methodology would also present an opportunity to adjust the algorithm to score Pedigree comics at the same premium as Signature Series comics. 

Link to comment
Share on other sites

Just letting you guys know that I have replied to my boss' email about 2 weeks ago.

I included the responses on here, and my own opinions on this subject.

Not sure if he has read it yet, I will follow up at our next registry meeting.  

 

Thanks!

-Brian

 

Link to comment
Share on other sites

I just stumbled onto this thread and want to add my 2 cents. I understand the opinions given but I worry about the lack of implementable solutions.  The complaint is that a small number of high scoring books skew set rankings, rewarding instant spenders with thick wallets instead of long determined work. As has been noted, the more skewed the book scoring is, the harder it gets to counteract.

Book scoring is based on fair-market value which incorporates desirability and supply/demand. Seems fair. Book scoring should only be determined by the book and not by set inclusion or any other factors. That keeps it simple. The sum of a collector's books should determine registry ranking. A set score could count a book at a higher or lower value.

Every set is different with various combinations of high value and low value books. A 50/50 split of points and completeness could work for some but not others. Another proposal would be to use fraction_complete * set_points. If a collector has one million point book in a set of 100 books, he would score (1000000) * (1/100) = 10000, and he could basically double his score by adding one book of any lowly grade. With a 50/50 split he would score just over 500000, and adding one more lowly book would be a small improvement.

A different approach could be to reward points and completeness in separate awards, but I suspect they need to be mingled by some formula here as well. Quality and quantity should both matter.

Set scores could also use something like 1+log10(points) for each book. This would vastly cut down the spread, since a million point book would only count as 7, while a single point book would count as 1. The spread here could easily be modified to whatever desired.

I'm mixed about this whole topic. I admire expensive sets but I also admire complete sets of good quality. I don't want to devalue the hobby.

Link to comment
Share on other sites

On 11/13/2021 at 5:43 AM, Rosland said:

I just stumbled onto this thread and want to add my 2 cents. I understand the opinions given but I worry about the lack of implementable solutions.  The complaint is that a small number of high scoring books skew set rankings, rewarding instant spenders with thick wallets instead of long determined work. As has been noted, the more skewed the book scoring is, the harder it gets to counteract.

 

Thank you for your input@Rosland

I definitely agree.  This issue has been in my head since I started.  Tying scores to FMV is a great idea, on paper.  Implementation is another matter entirely, due to the sheer volume of books in the registry.  I will be taking up this cause again after the new year,  I would very much like to implement an automated solution when it comes to scoring, and see what it will take to implement a completion modifier.

Link to comment
Share on other sites

I have the 7th Best Batman 1-100 CGC Registry set. I think there should be extra points awarded for a complete set with photos. A few single high grade books shouldn't beat a complete set.

I also think sets that are broken up and sold should be put into an Archive Collection category. Everyone does not register their collection but we know when a collection is sold.

The number one Batman 1-100 set has been broken up and sold. It should be "retired" not still competing. Maybe a Hall Of Fame Collection category needs to be set up for these collections.

Link to comment
Share on other sites

@wytshus

I mean a run is all good, but if the argument is "affordability?"

What is the difference in buying a bunch of $200 keys, rather than Buying a bunch of $200 in high grade non keys to fill up a run?

I get that high grade may "skew" but that shouldn't offset that there are still a lot of affordable out there (thumbsu 

Imo it should matter the set only :cheers:

Yes it will cost a lot to fill a set of Batman #1-100? But less to do a run of batman #200-300 and so one to #300-400

I think the scoring is fine, just more sets should be considered for awards? Rather than the highest ranking?

Idk how you would determine, other than description and pics. I would be just as :cry: if the awards were due to movie buzz? I mean there is no logical progression other than completeness, pics, and description, which I "thought was already in place"

The only thing I can concretely determine, is maybe there should just be more awards? To give more people a chance? I don't want to single me out, but I feel I'm being singled out due to: I can't afford to spend $200 on run of the mill Batman books from #300-400 that most I've never read, 100 books at $200 is still $20,000 that you could have spent on a set earlier in the run. I respect run collectors, and believe they should be rewarded with a completion bonus. 

But just because the "run is 'more affordable', which it isn't in the long run", maybe more awards available to pick more sets, and or, more sets be broken up so runs are easier to complete?

I'd rather spend $20,000 On an AF #15 than a bunch of Batman books that would earn me $500 in grading credit

in the long run.

Maybe I'm being singled out, though I do own runs, so in my defense I'm ignorant or not understanding the logic here :shy: 

Link to comment
Share on other sites

The only thing I can think of is:

If it costs $20,000 to complete batman#300-400.

And it costs $20000 for a ASM#1 or af #15.

Then the points and completion bonus should more correlate!

If I'm given $20000 to buy a run of batman I may I may not, but if you're thinking I'd do it to win a registry award, then thats even more love I'd have to have for Batman 

Over a $20,0000 dollar car and etc.

I'm not that rich either.

That's the same thing that skews imo

If a high grade ASM #1 skews then so does $20,000 worth of batman.

Y'all bout to price me out 😂 @wytshus

Link to comment
Share on other sites

Thanks for your input @ADAMANTIUM

 

Great points, and I have brought them up repeatedly at our weekly meetings.

Thing is, the points are pretty much for entertainment purposes.  When we pick the award winners, we look for way more than points.  The winners I picked last year didn't have insane point totals, I looked at the comments for passion and sincerity.  There were a lot of sets with cut and pasted summaries, no mention about how they felt about the books, nothing about how they acquired them.  

Keep in mind, I had just hired in, and wasn't really up to speed at the time, I had to rely on my colleagues to know what to look for.  When I took over the registry, I got kinda freaked out about points, and I'm still a little freaked about the fluctuations over the past 2 years. 

There are going to be some major updates to the registry this year, and I have insisted that we take a deep dive into how keys are scored, and how/when they are updated.  The speculation market is insane right now, and I don't think that it's worth our time chasing these fluctuations when we don't even set prices in the market. 

This issue is pretty much always at the back of my mind, especially when I do score adjustment requests.  It's still the hardest part of my job, and it turns out, it doesn't really have to be.  It's just the way I am, I love challenges!  lol.

More to come.

-Brian

 

Link to comment
Share on other sites

On 4/5/2022 at 1:08 PM, wytshus said:

Thanks for your input @ADAMANTIUM

 

Great points, and I have brought them up repeatedly at our weekly meetings.

Thing is, the points are pretty much for entertainment purposes.  When we pick the award winners, we look for way more than points.  The winners I picked last year didn't have insane point totals, I looked at the comments for passion and sincerity.  There were a lot of sets with cut and pasted summaries, no mention about how they felt about the books, nothing about how they acquired them.  

Keep in mind, I had just hired in, and wasn't really up to speed at the time, I had to rely on my colleagues to know what to look for.  When I took over the registry, I got kinda freaked out about points, and I'm still a little freaked about the fluctuations over the past 2 years. 

There are going to be some major updates to the registry this year, and I have insisted that we take a deep dive into how keys are scored, and how/when they are updated.  The speculation market is insane right now, and I don't think that it's worth our time chasing these fluctuations when we don't even set prices in the market. 

This issue is pretty much always at the back of my mind, especially when I do score adjustment requests.  It's still the hardest part of my job, and it turns out, it doesn't really have to be.  It's just the way I am, I love challenges!  lol.

More to come.

-Brian

 

awesome, and believe me that is encouraging!!!! Nice perspective, I'll stay tuned :) 

Link to comment
Share on other sites

It would be a ton of work, but maybe each competitive set needs its own calibration of points, potentially loosely tied to FMV/Rarity with some type of completion bonus but done in such a way that Keys are given their due weight without swallowing the whole points total.
 

I suppose you could start with a formula and then begin tweaking sets over time based on requests in a “Fix the Points of this Set” thread. 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
3 3