There's multiple strategies to prevent this, especially considering the layout of the map.
1. On your island, create walls around it. This will prevent them from knocking you off, if the wall is high enough.
2. On their islands, quickly avoid them and go into one of the 2 mushroom buildings...