"Love thy neighbor as yourself; but don't take down the fence." -- Carl Sandburg
This one-time voice of America obviously had a funny take on the second commandment and he makes a great point. Love your neighbor, but watch your backyard.
The operative word he uses is fence. Fences are easily looked over the top of or through to see how your neighbor is doing. Fences have gates that are easily opened to let your neighbor in, or bid them good night.
The poet isn't saying to build a wall so as to isolate yourself. I suppose that would be the best way to stay completely safe, but who wants to live as a hermit?
It could be your neighbor to the left or right, the north or south. Who knows how great a neighbor they could be?
It's certainly okay to exercise caution, but when it's appropriate, open the gate! Love them as yourself and prosper together!!