You're right, to a point. If they are indeed wrong, then the onus is on you to explain their fault in a respectful and non-dickish (example: telling them to pound sand is the opposite of non-dickish) way. For instance, when I first bought my house my neighbor was cutting swaths of my lawn by not picking up the damn mower when he turns to do another pass, and instead walking over my lawn. This wouldn't have been a problem if he actually finished the strip (it was just a little 4x4ft box in front of the sidewalk) but instead he left semicircles in my yard. Politely explained where the property lines were, and we don't have a problem anymore. Conversely, if the OP is wrong, then they need to take reasonable steps to correct this. Up and moving the fence on a moment's notice is pretty unreasonable, but moving the playset can be done with minimal effort.