My wife and I watched the show from the beginning for the first 3 seasons and then we just couldn't take it anymore. The CW "teen drama", "teen angst" aspect of the show was just really boring/hairpulling for the both of us. There was a lot of that stuff (I love her/him, does she/he love me, I can't tell her/him this because of some silly reason, etc.) in the early seasons.
I then came back to it in its last season and I was happy that I did. I really enjoyed the last season. It focused a lot more, imo, on the superhero aspect/ comic universe that I felt was lacking from the first few seasons. When I revisited the seasons that I had skipped, I realized I didn't miss much of the overall story, just a few cool moments here and there.
Smallville wasn't a horrible show, and it can keep your attention for the most part, but it's not mindblowing by any means. It has a very different feel then the CW hero shows that are playing today.