Welcome to the Forum, @jepmus! 
What you’re describing seems to be a persistent issue with the Google Home app on iOS. This topic might provide a little more insight:
Even with the Google Home app on Android, I get inconsistent performance from my Wyze cameras. Not only that, I have some non-Wyze cameras connected to Google Home that don’t stream in the app at all but have no apparent difficulty streaming to Google Home-connected smart screens (Chromecast HD, Google TV, and Home Hub).