Based on how you’re describing the issues your experiencing, I suspect non-pan/tilt cameras would be better suited to your application, as you suggested. Although Cam Pan v3 has the Detection Zone feature, I have never used it because I’ve placed that model of camera indoors only and have Track Motion enabled. When the camera is allowed to move and change its view, setting a Detection Zone isn’t much use, I don’t think.
Reportedly, setting a Detection Zone even with motion tracking enabled used to have some utility by having the effect of setting a “home” Waypoint for the camera to return to when otherwise idle, but it’s been described elsewhere in the Forum that recent updates (possibly app; likely server-side) have caused this behavior to no longer work that way. If you’re using Pan Scan (you mentioned “[seeing] two sides of the house via waypoints”, but I don’t know if you’re using that feature), then a Detection Zone wouldn’t be very useful for that, either.
Even with stationary cameras, though, my experience has been that setting an effective Detection Zone requires some trial and error, especially living in a windy area where the cameras can potentially see trees and flags. Even tree shadows can be problematic.
Your issues with motion tracking (having a camera pointed at trees and suggesting a solution of physically masking part of the view area) remind me of a couple of relevant Wishlist topics, in case you’re interested in voting and commenting on those:
I like Cam Pan v3, though I’ve been disappointed with some recent changes, and I think there’s definitely room for improvement and features that I believe could be implemented via firmware and app changes.