Give it back to him.
The agreement was based on you paying him in full when you received your returns, basically counting your eggs before they hatched. You didn't get the refund you expected and are unable to pay him in full as you agreed.
He went on the trust system, and unfortunately, you didn't keep up your end of the agreement. Return it to him with your apologies.
That's probably not what you wanted to hear, but that's the reality of the situation.