That was my initial experience with Smart Focus on the first Cam v4 I set up, too, but I eventually played with it some more and described my first experience seeing it work in another topic:
It’s kind of a nifty feature, but I haven’t gotten much actual use out of it. I figure if Edge AI that’s in Cam v4 is there to support Smart Focus, then it should actually be doing something (I’d consider to be) more useful and provide on-board Person Detection.
I don’t know if a different Internet service provider will resolve your problem, but it might be worth a shot, especially if it means you’d get a different router/wireless access point that could address a potential Wi-Fi bottleneck or other issue as @dave27 previously suggested.