Bulkhead seat and people using it as a hallway?

Sitting in a bulkhead seat in a 767-400 long haul flight and people keep cutting through to change aisles/get to a bathroom on the other side (there are two aisles and seating is a s 2-3-2 configuration)

Am I supposed to be cool with people stepping over me or making me move my legs when there are bathrooms in the aisle they walked up? It’s a premium seat that we paid for.

Not sure if I should be annoyed or not (actually, I AM annoyed but aren’t sure if it’s justified)