In my experience, building a fence around your settlement pretty much protects it from random visitors.
For instance, if you release a Companion within your fenced settlement, you pretty much have to guide him through a fence to get him to leave. Even though he will readily climb the fence when following you as your Companion, he seems to forget his climbing skill as an independent NPC.
If he has an ax, he will fell large trees you have incorporated into your fence, and then leave.
I've had wandering hunters and woodsmen stalk the area around my settlement for days, and even weeks, without ever crossing the fence. I've had foreign traders hover around my fence for several days before leaving the area, without ever crossing.
The only travelers that have ever willfully crossed my fence was when a band of 4 robbers attacked and made life interesting.
I guess the rule is, good fences make good neighbors, eh?