Yes, it's because of the sillier covers. And it's not a title that had a lasting run, like Batman, Superman, Action or Detective, so there isn't even that extra incentive to buy them to complete a run. It does feel like fluff, something extra not overly integral to the characters' publication history, and thus not overly integral to a collection of said characters.
The first appearance of the Scarecrow doesn't have as much as a demand as his second appearance in Detective because in this world of CGC, all you see is the cover and this ain't the cover you have in mind for a key Scarecrow book: