• When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network.

Is it just me or are the boards a hot mess of 500 Errors?
5 5

Started by Point Five,

267 posts in this topic

universal soldier

universal soldier

Posted
Posted
On 9/10/2021 at 2:46 PM, sckao said:

I got that error too... and this seems to be the most common issue: server misconfiguration or local DNS issues. (localhost). Possibly exacerbated by load issues I guess or memory issues if the time of day and the site going down is a pattern. We've had issues with growing/runaway database tables as well that frequently took down sites. Adding new grading categories like cards, etc. must be adding new strain on their Board software/system.

Some of the server guys are absolutely brilliant and can help you diagnose these issues in a heartbeat. I really hope someone will look at these errors and resolve the issue. The traffic that comes to these forums only help with brand awareness and improves google search results for CGC. It benefits them to keep this place up and running.

Link to comment
Share on other sites
ADAMANTIUM

ADAMANTIUM

Posted
Posted
On 9/10/2021 at 4:38 PM, grendelbo said:

I think it was 2 months ago today that the boards came back online after Blackout Wednesday.

Ya 2 months ago today.... around my birthday 

AssuredVerifiableDuckling-max-1mb.gif

Link to comment
Share on other sites
sckao

sckao

Posted
Posted
On 9/10/2021 at 1:02 PM, universal soldier said:

Some of the server guys are absolutely brilliant and can help you diagnose these issues in a heartbeat. I really hope someone will look at these errors and resolve the issue. The traffic that comes to these forums only help with brand awareness and improves google search results for CGC. It benefits them to keep this place up and running.

True. These boards are perhaps their most public/best outward facing branding mechanism. There's probably a huge backend system we're not seeing right now linking their massive growth in their Florida offices with what we think are their online web presence. Every new employee, new workstation, new scanner, bar coder, etc. that they add to their systems to handle the ever growing demand, plus the constantly growing data black hole these boards must represent in terms of data storage, increases the strain on their system. Every time someone uploads an image, that image is now being hosted by CloudFront which indicates that they're using Amazon's AWS service to host the ever increasing number of attachment all of these boards must generate.

(And the CGC Boards are just one of many being run by the parent company.) While they are offloading the actual data storage and caching, the links to those items are still being maintained by the board's database software which must also be integrated with their registry databases, their blog databases, their internal grading databases, pressing databases, accounting databases, etc. Each time, multiplied by each Collectible industry (Comics, coins, stamps, cards, video games, etc.)

So it's not all that surprising that generating an entirely new parallel structure for their new videogame category, for instance, might break some of their already calibrated systems that were already near some sort of maximum in terms of load or result in a misconfiguration. We just don't see that yet publicly.

Link to comment
Share on other sites
Mutie Fan

Mutie Fan

Posted
Posted
On 9/10/2021 at 3:16 PM, sckao said:

True. These boards are perhaps their most public/best outward facing branding mechanism. There's probably a huge backend system we're not seeing right now linking their massive growth in their Florida offices with what we think are their online web presence. Every new employee, new workstation, new scanner, bar coder, etc. that they add to their systems to handle the ever growing demand, plus the constantly growing data black hole these boards must represent in terms of data storage, increases the strain on their system. Every time someone uploads an image, that image is now being hosted by CloudFront which indicates that they're using Amazon's AWS service to host the ever increasing number of attachment all of these boards must generate.

(And the CGC Boards are just one of many being run by the parent company.) While they are offloading the actual data storage and caching, the links to those items are still being maintained by the board's database software which must also be integrated with their registry databases, their blog databases, their internal grading databases, pressing databases, accounting databases, etc. Each time, multiplied by each Collectible industry (Comics, coins, stamps, cards, video games, etc.)

So it's not all that surprising that generating an entirely new parallel structure for their new videogame category, for instance, might break some of their already calibrated systems that were already near some sort of maximum in terms of load or result in a misconfiguration. We just don't see that yet publicly.

I have no idea what you said, but you convinced me. (thumbsu

Link to comment
Share on other sites
universal soldier

universal soldier

Posted
Posted
On 9/10/2021 at 6:09 PM, Here Oh said:

Except when CGC doesn't want to improve the search results.

Screenshot_20210910-180655_Chrome.thumb.jpg.4b5677166d6f526642484e4281aa6970.jpg

That's interesting, looks like they are preventing the boards from being indexed. If you google search the site --> site:www.cgccomics.com/boards it only brings back the coin forums. (shrug)

Link to comment
Share on other sites
universal soldier

universal soldier

Posted
Posted
On 9/10/2021 at 6:16 PM, sckao said:

True. These boards are perhaps their most public/best outward facing branding mechanism. There's probably a huge backend system we're not seeing right now linking their massive growth in their Florida offices with what we think are their online web presence. Every new employee, new workstation, new scanner, bar coder, etc. that they add to their systems to handle the ever growing demand, plus the constantly growing data black hole these boards must represent in terms of data storage, increases the strain on their system. Every time someone uploads an image, that image is now being hosted by CloudFront which indicates that they're using Amazon's AWS service to host the ever increasing number of attachment all of these boards must generate.

(And the CGC Boards are just one of many being run by the parent company.) While they are offloading the actual data storage and caching, the links to those items are still being maintained by the board's database software which must also be integrated with their registry databases, their blog databases, their internal grading databases, pressing databases, accounting databases, etc. Each time, multiplied by each Collectible industry (Comics, coins, stamps, cards, video games, etc.)

So it's not all that surprising that generating an entirely new parallel structure for their new videogame category, for instance, might break some of their already calibrated systems that were already near some sort of maximum in terms of load or result in a misconfiguration. We just don't see that yet publicly.

I didn't think about all the internal aspects. Some really good points here. When you look at the over-arching infrastructure that they support I'm surprised that they allow attachments to be stored within the system like you said, especially given the amount we use them here. It must be a massive S3 bucket in place. Clearly they aren't using the elastic option for the boards :insane:. I can relate though, our lower environments get shut down over the weekend to save some pennies.

Link to comment
Share on other sites
sckao

sckao

Posted
Posted
On 9/10/2021 at 5:10 PM, universal soldier said:

That's interesting, looks like they are preventing the boards from being indexed. If you google search the site --> site:www.cgccomics.com/boards it only brings back the coin forums. (shrug)

I think they are having a specific issue with Google actually.

Bing.com seems fine.

site:cgccomics.com Grading Contest

are the parameters I used.

 

Link to comment
Share on other sites
universal soldier

universal soldier

Posted
Posted
On 9/10/2021 at 8:23 PM, sckao said:

I think they are having a specific issue with Google actually.

Bing.com seems fine.

site:cgccomics.com Grading Contest

are the parameters I used.

 

This explains it: https://www.cgccomics.com/robots.txt 

user-agent: *
disallow: /boards/*
Link to comment
Share on other sites
adampasz

adampasz

Posted
Posted (edited)
On 9/10/2021 at 5:19 PM, universal soldier said:

I didn't think about all the internal aspects. Some really good points here. When you look at the over-arching infrastructure that they support I'm surprised that they allow attachments to be stored within the system like you said, especially given the amount we use them here. It must be a massive S3 bucket in place. Clearly they aren't using the elastic option for the boards :insane:. I can relate though, our lower environments get shut down over the weekend to save some pennies.

S3 storage is cheap. Like 2¢ per gigabyte per month.

At that rate, 5,000 users could each store 100 high-res images for ~$20/month all together. (10,000 users X 1MB X 100 images = 1,000,000 MB = 1,000 GB X $0.02/mo)

AWS hosting costs can add up quick, but -- unless you're storing HD video -- the bottleneck is probably not due to S3.

https://aws.amazon.com/s3/pricing

My guess is, the software and DB schema for the message boards weren't designed to support the sheer volume of posts that have accumulated over 20 years. If basic tasks, like retrieving posts, require complex queries across multiple tables, it can reach a tipping point where performance degrades exponentially. Simply throwing more hardware at it may not solve the problem.

I know there was a software upgrade a few years ago, but maybe that was just a temporary solution.

Edited by adampasz
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
5 5
Go to topic listing