Unless you discussed it at the beginning, I would think if he doesn't buy the deposit should go back.
I would only take a deposit from now on if you tell them that this is you letting them get the money together, so if they don't and waste your time it is non-refundable. Otherwise, what really is the point of a deposit?
My 2 cents.