I took out those little foam cut outs and tossed them. Then put the speaker in there. I wish I wore that helmet to work today, I'd take pictures. But again, I'm on a slightly different model than you. SMH10 vs. 20
I can even wear ear plugs. Depending on how far I shove them in my ear they will sometimes rest against the speaker. That may help you since you said it bothers your ear. But then its a trade off of a very slight pressure of the foam plug pushing slightly against your ear canal (really not a huge deal since there is very little pressure, at least for me.)