We took our son at 3 and he doesn't remember anything. We just took both our son and daughter in January (now ages 9 and 6) and they didn't like it at all. I won't be going to Disney again.
Like others have said, save your money and take them when they are older. Also don't go in July, you will die from the humidity.