The very first comic store I went to was the Comic Kingdom run by Bill Cole. I think it was only open on specific days, and it was in some weird industrial type building. (The Colonial Village Mall in Quincy, according to Mackenzie999.) I wish I could remember the specifics of the location, but it was 40 years ago and my folks are gone and my memory of that aspect is a little hazy. I suppose I could just email him.
The comics were on staggered/tiered wall units, I don’t recall going through boxes.
Walking in and seeing those old books from the 60s, which at the time seemed as long ago as the turn on the century, was amazingly magical. I bought my first back issues there: Marvel Tales 1 (recommended by Bill for the young reader) FF 48-50, ASM 44 and 45. They remain in my collection, except for the FF 49 and Marvel Tales; I think I sold or traded those as a kid. I also got my Origins of Marvel Comics and Son of Origins there; they’re dog-eared now but still on my bookshelf.
From Bill I found out about the local monthly show, The Sunday Funnies at the Boston Howard Johnson's and run by Don Phelps. The monthly show spoiled me a bit. When more comic shops opened up in the ensuing years, I didn’t find any of them a good source for back issues – you could always get much better deals and selection at the show.