My heart screams that I should do exactly what you did, and then, I remembered someone else who posted something similar a few months back. So, I have re-thought about it.
In a situation like this, I would have to tell myself not to do it. He knew he didn’t know what it was worth. He did not try to look it up, and you are not a dealer with whom he should have a legitimate reason to expect a fair price. He also did not operate under a mistaken factual belief, for example, getting the artist’s attribution wrong, which you took advantage of. In a sense, he played you— you did his homework for him and suffered for it. Once he made an offer of $500 of his own free will, you should have paid him in cash, gotten a receipt, and left. He would have been happy with it and so would you. Yes, hard to do, I agree.
And yes, you probably did have a binding oral agreement at $4k. But you would be stunned to hear what he would actually say at trial after he spoke to a lawyer about it.