Critique My Site: ComicSeeker.com

[innocuous]

I've also pondered how to solve this problem of finding books across a large number of sites. Some sites have a notification service, but it runs only once a day (if ever) and I still miss some books from other lucky collector/sniper. In addition to some very specific search/parsing technology that comicwiz mentions, I thought about a method of creating web services that comic web sites could utilize to notify and communicate new and changing information so collectors could literally get the latest up to the minute information regarding inventory availability.

 

This is interesting and leads to an alternate approach. As the "consumer" of comic books, it is the dealer who should be jumping at the chance to sell the book to you...and you not having to lower yourself to finding the book at ComicSeeker.com.

 

So, what if you could register on a website your current wantlist. Then, whenever a dealer got in new stock, they could compare against the wantlist of everyone on the Internet who posted their list. This way, before the dealer even posts his new books online, he could send you a message that the book was now available.

 

You would think making a quick sale on the book would be enough motivation, but...

[Aman619]

Dont many sites already say they do this now?

[comicwiz]

I've also pondered how to solve this problem of finding books across a large number of sites. Some sites have a notification service, but it runs only once a day (if ever) and I still miss some books from other lucky collector/sniper. In addition to some very specific search/parsing technology that comicwiz mentions, I thought about a method of creating web services that comic web sites could utilize to notify and communicate new and changing information so collectors could literally get the latest up to the minute information regarding inventory availability.

 

This is interesting and leads to an alternate approach. As the "consumer" of comic books, it is the dealer who should be jumping at the chance to sell the book to you...and you not having to lower yourself to finding the book at ComicSeeker.com.

 

So, what if you could register on a website your current wantlist. Then, whenever a dealer got in new stock, they could compare against the wantlist of everyone on the Internet who posted their list. This way, before the dealer even posts his new books online, he could send you a message that the book was now available.

 

You would think making a quick sale on the book would be enough motivation, but...

 

I think what sterling is bringing into the discussion is this fundamental debate around search engine technology and the notion of intrusion, and its a debate that looms like a dark cloud over innovators of search technology.

 

What's interesting about the debate surrounding search technology in current times is the duality of perception, as the technology is looked at both with enthusiasm and condemnation.

 

Enthusiasm on the level of possibilities it can provide in areas such as homeland security and monitoring/tracking the activities of sexual predators, just to name a few. Until now, enforcement authorities have had to use hit-and-miss methods such as cyberbaiting to track these forms of criminal activity. The problem is that cyberbaiting on the web is a lot like trolling the vast open sea with a 10-foot fishing boat.

 

It is no coincidence that when the US Dept of Justice asked google to hand over "trade secrets" on its search engine technology, that behind its "good-will" nature of seeking "cooperation" was this imperative seeking their aid on security matters. This idea of using search technology to combat such things as child pornography as well as the possibility of intercepting communications between terrorist groups were initiatives that could assist governments in areas it was ill-equipped in handling using their current methods.

 

But its the same courts, lawmakers and enforcers of law that are also pointing their fingers at companies like google, condemning such search engine innovators of digging-up information that is infringing on copyright, peoples privacy and a wide number of other levels of personal, corporate and government intrusion. A number of these cases are in the courts, and will be for some time, and it is doing nothing more than creating an obstacle for innovators to take search technology to the next level.

 

Here's how I see it. The days of innovators such as Tessla are long gone. His was a form of pure innovation, driven by passion, and less to do with the commercial viability of their contributions. And in a way, its a good thing that innovators are taking what they can get nowadays. Times are gone for honest men, and the legacy of squatting on innovations for the almighty dollar are happening less and less because innovators are not so quick to give away their work.

 

Cases such as Tessla's inform us of the shrewd and sometimes dehumanizing approaches of governments and corporations to snatch away innovation for their own greed and self-interested purposes, with little or no acknowledgement at all to the people who first put things down on paper and made them reality.

 

So while the military were the first to bring the web to life, I'm certain that they are biting their tongues over the fact that they weren't able to be the principle innovators of search technology. I also think that because search technology is primarily in private ownership, its created a sense of insecurity and and an air of mistrust that I believe might not have existed at all if search technology ownership was in government hands.

 

And as governments and their officials push search engine innovators to a point where they are demanding search engine companies to "cooperate", their demands are being made at the expense of intruding on the intellectual property of the search engine technology owners. A demand that stands to do nothing more than disturb the waters of future innovation in combating such things as child pornography and homeland security.

 

The main problem with the bullying approach being used by governments is that there isn't a large supply of developers in the area of search engine technology. You upset the few that are in the know, and you throw away the opportunity of ever getting them to assist in combating criminal activity that occurs online.

 

I think the real question to be posed: who is actually doing the intruding, the innovator or the followers of the innovation?

[sterlingcomics]

Dont many sites already say they do this now?

 

I'm not aware of any...at least do it on a level that I'm interested in.

 

Pick up a crack pipe with me...

 

You sign in to a service, place in the want list....it immediately queries all participating sites (a very CPU/traffic intensive process as comicwiz indicates)....and you get books that meet your criteria.

 

Imagine having a notification alert running in your system tray....whenever a book INSTANTLY arrives in a participating sites inventory...it sends the messasge to the service...compares it against the want list...and if it is included, sends you an immediate e-mail or instant message or notification to your system tray.

 

Now imagine that you could buy it from that same interface....never really using the vendor's web site....kinda like Amazon Shopping on steroids...this would be a very useful tool to increase sales from compulsive buyers (kinda like me).

 

I call it the Comic Star Alliance.

 

OK, put the crack pipe down.

 

The concerns that comicwiz talks about are also my worries. My concerns over this approach includes the fact that the service must be "trusted" in order for this to be effective. Sure, you can encrypt the message similar to a credit card transaction, SSL it etc...but then there has to be general trust in the process. Just from over 17 years of IT experience I know people want to feel like they are in control...this approach takes away some control...but adds incredible flexibility and features that I dream about having as a comic collector.

 

I think it can be done and sold...but the market is so niche that I think there would be extremely limited returns on doing such a project. Perhaps a more reasonable approach might be effective....hey guys....if you ever want to discuss perhaps an opensource project in a collaborative manner I'd be interested.

[comicwiz]

You sign in to a service, place in the want list....it immediately queries all participating sites (a very CPU/traffic intensive process as comicwiz indicates)....and you get books that meet your criteria.

 

Imagine having a notification alert running in your system tray....whenever a book INSTANTLY arrives in a participating sites inventory...it sends the messasge to the service...compares it against the want list...and if it is included, sends you an immediate e-mail or instant message or notification to your system tray.

 

Now imagine that you could buy it from that same interface....never really using the vendor's web site....kinda like Amazon Shopping on steroids...this would be a very useful tool to increase sales from compulsive buyers (kinda like me).

 

I call it the Comic Star Alliance.

 

Not to toot my horn, but about 75% of what you're suggesting could be easily handled through the FetchComics system. The other 15% would involve tweaks, and interface development. The backend intelligence is already all built.

 

The hardest part of such an endeavour would actually be in carrying forward the business logistics, and mainly to get buy-in from participating vendors. Not that its impossible, but you would have to corral all the comic shop owners to join-in and keep a high-level of consistency in supplying new inventory, and keeping their inventory up to date.

 

For instance, one basic requirement would be that each comic shop (online or brick & mortar) needs to have some form of digitally indexed inventory (preferably already online, and already searchable). For those that don't, there would require a ramp-up process and a nominal dollar cost associated (ie. membership fee).

 

I've done this type of thing with portal sites, so its not out of ther realm of possibility to get this kind of buy-in. The will has to exist on all levels. The trick is to get financial tie-in, so that participating sites have a stake in the evolotion of the project, which in turn will make it less likely that they're interest and responsibilities will gradually wittle away over time. What I mean by this is that most comic shop owners actually eBay because of the low-payoff in updating their own sites. With feeBay, at least you're likely to get some traffic. Online shops tend to be a little on the mid-range scale of development, and generally users are turned off when things like search capabilities aren't offered to track down books in their inventory.

 

Recognizing of course that high-payoff activity keeps the ship afloat, the one fundamental error a great many online shops make is not recognizing the importance of having an inventory on their own site. It might not translate into high volume sales, but its a very important "sticky" element that can not be ignored or overlooked. Obviously, if its searchable, great. But even if its an excel file that is converted to html, and I can scan it by alpha, and its updated regularly, I'll go back and check from time to time.

 

An incentive has to exist for them to not only join-in initially, but to also upkeep their list of inventory. The last thing you want is an elaborate system like the one you speak about turning up listings that have already sold. On the one hand, you can't blame the technology that is finding it, but rather, the culprit in this scenario would be that the inventory being searched on is not being kept up to date.

 

It certainly sounds interesting enough that if something were to be put down on paper, and enough vendors would "join-in" that I'd want to get involved in such a project.

[sterlingcomics]

The hardest part of such an endeavour would actually be in carrying forward the business logistics, and mainly to get buy-in from participating vendors.

 

Yeah...the development of said service wasn't what I was referring to in terms of the crack pipe...it was getting the buy in of vendors to make it actually a worthwhile venture.

 

The last thing you want is an elaborate system like the one you speak about turning up listings that have already sold.

 

That was built in to the prototype of the web service...if the item sells...a notification gets built that will remove it from want lists, etc. Although--if it doesn't sell through online venues it will depend on the accuracy of the vendor to keep inventory up to date (as you previously mentioned).

[comicwiz]

That was built in to the prototype of the web service...if the item sells...a notification gets built that will remove it from want lists, etc. Although--if it doesn't sell through online venues it will depend on the accuracy of the vendor to keep inventory up to date (as you previously mentioned).

 

Sounds like a solid model.

 

Perhaps its only flaw is that it sounds a bit ambitious because in order for that type of interfacing to occur flawlessly between varying data types, you would have to either repopulate one central repository with all the vendor inventory listings, or require each vendor to recreate a listing for each book in their inventory.

 

At its most basic level, this is the only way you would get rules established and policies that would work with what would otherwise be a wide variety of differing data types. The idea of automating the system to notify, and remove listings as they sell is best (and likely only) handled through the task of re-inputing each item. For vendors with thousands of books in their inventory, such a task would be too onerous and time-consuming.

 

You might find an alternative in tapping into the existing data inventories in whatever formats they are currently being made available, and try to establish checks that would cross-reference listings against any order or invoice record, but even then, the best way and only sure-fire way would be to have to the listing permanently removed.

 

There is no doubt that you could do such things like flag them with a "sold" icon, or not have them appear any longer. Its just a lot more tedious and can get out of control in terms of resources required to run these different filters and checks. Its an approach wrought with problems as its likely to bog-down processes more and more as neglected listings start piling up instead of getting yanked.

[Timulty]

Actually, this isn't as difficult as it seems. Again, buy in by vendors is important, so you can have access to their files but setting up automated processes via a flatfile type setup, both on the export and import ends is not difficult with the existing software around. I have built such systems and once you have designed one, you can design them all. Just need access and a reliable processes developed. Bad part about this is latency. You will probably only import and export said files once a day. This might be too much latency for the intended application.

[comicwiz]

Actually, this isn't as difficult as it seems. Again, buy in by vendors is important, so you can have access to their files but setting up automated processes via a flatfile type setup, both on the export and import ends is not difficult with the existing software around. I have built such systems and once you have designed one, you can design them all. Just need access and a reliable processes developed. Bad part about this is latency. You will probably only import and export said files once a day. This might be too much latency for the intended application.

 

Yes, I agree that flatfile formats would bypass a lot of the technical hurdles associated to bridging together multiple data formats, but latency and system bog-downs would still be an issue, and lets not forget the amount of time and effort required to massage the data.

 

Its my professional opinion that In order to leverage existing data sources available at -- lets use ComicLink, Metropolis, Mile High, Harley Yee and my site as examples -- the ideal way to search on them is by leveraging current online catalog repositories by using a federated search technology able to plug into all the current data sources without the need to import/export, index or repopulate any data:

 

 

This is the only way to leverage their existing inventories in their current formats, search on them and return results in formats to be managed as desired and according to the portal model sterling describes.

[innocuous]

Other industries, such as financial services and newspapers have been working on such agreements. I don't know much about XML/RSS feeds, but that seems to be the solution. Of course, if vendors agree to publish their inventory via XML/RSS.

[sterlingcomics]

You will probably only import and export said files once a day. This might be too much latency for the intended application.

 

Yes it would...at least in my vision of the solution. For me, it would mean instant notification the second a vendor updated inventory. Instant notification = compulse buy = increased sales. I'd even do the modification for the first few vendors to test the waters.

 

Nice info comicwiz (I personally like PHP). I like the way you approach a problem! The main concern I have with querying web sites directly is the fact that there could be scalability problem as this would increase traffic to their sites. The other concern is the timeliness of the information, which was paramount in my vision. The overall architecture you provided in the diagram is great.

[sterlingcomics]

Other industries, such as financial services and newspapers have been working on such agreements. I don't know much about XML/RSS feeds, but that seems to be the solution. Of course, if vendors agree to publish their inventory via XML/RSS.

 

This would be a common communication vehicle to provide full and last incremental updates of inventory via this technology. However, this is the intrusive part....their current web site would need to generate this file during add/modify/delete events within their internal application.

 

In my mind, said service would need to keep inventory of third party sites (so vendors don't take a hit in traffic with constant queries) and synchronize when inventory changes. URLs could be stored to any images the vendor has on the book in question.

 

Thanks guys for waking up the beast on this. I've literally been thinking about this for the past three years. It's even replaced my personal web site in terms of priority. I'd love to consult with you guys as I continue down this path.

[sterlingcomics]

Perhaps its only flaw is that it sounds a bit ambitious because in order for that type of interfacing to occur flawlessly between varying data types, you would have to either repopulate one central repository with all the vendor inventory listings, or require each vendor to recreate a listing for each book in their inventory.

 

This is where I think web services, stored procedures and/or identity management solutions could help solve this problem. After an initial "load" of the vendors inventory...it is kept up to date via communication between the vendor and the centralized repository...receiving messages as items are added/modified/deleted. The communication could come from either web services (optionally executed from a db stored proc) or an event through the identity management system. Because each vendor has different tables/inventory etc...there would need to be some data mapping involved.

 

I do identity management projects all the time where we map pieces of information between disparate systems so this type of integration is cake for me. Actually, with my knowledge of identity management solutions I could actually do the integration without any code changes on the vendors end....looks like I solved the intrusive problem.

[innocuous]

 

Family plans got cancelled for this long weekend. So, since I've received so many link requests, I created a link index. It's light on the links right now, so check it out and add your favorite links...please. And, feel free to link to me.

 

If you've posted to this thread, I've already added your link. Let me know if you want to change the description.

 

http://www.comicseeker.com/

 

Click the big red link in the middle of the page that reads: ComicSeeker Web Directory and Link Index

 

Next up is t-shirts, so that I'll have something to wear to SD.

[Timulty]

Looks pretty good. I think, that site will really start taking off now. Great way to search for your favorite books.

[innocuous]

Thanks for the compliment. Also taking requests for features.