I applied 220.127.116.11 to several Wyze Pans this morning and they no longer work meaning, I cannot connect to them in the app - the app doesn’t even get to “Authenticating 2/3”, My other Wyze Pans on 18.104.22.168 are still working perfectly.
What I have observed - the cameras connect to Wifi, they get DHCP, I can ping their IP, I can see their outbound traffic to NTP, DNS, various IPs at port 443, 80, 8443. So they are alive, but not useful.
My first thought is that I run a DNS filter and perhaps the firmware is connecting to a new DNS name that is being blocked, but I see no DNS being blocked on the DNS filter for the cameras.
Has anyone else experienced this? Any thoughts on how to get them back?
Some more notes. It seems directly flashing 6.156 via an SD card also bricks the cameras. I also noticed that if using the app to update, it updates to 6.143 first, reboots, and then gets stuck. Now, it may well complete the update to 6.156, but as we see, 6.156 just doesn’t work. I even tried regenerating the QR code and all that. It reads the QR code on 6.156 but then can’t connect to the WLAN. So yeah, if anyone is listening, 6.156 has some issues.
22.214.171.124 is no better and completely bricks all of my Pan Cams. Happily, I emailed on my existing support tickets and they have been closed with no resolution, notification, or explanation. Seems this is a black hole issue. :-/
I ran into similar issues since 6.156 and 6.193 with my 4 Pan Cams and had to keep reverting to 5.111. I decided to spend a little more time working on the issue today and I was finally able to successfully upgrade the firmware on all my cams. I have NTP redirection configured on my OPNsense firewall and had it configured with Asus-merlin before that. As soon as I disabled the NTP redirect the new firmware worked fine. It seems like the firmware 5.111 and before had no issue with NTP redirect but anything newer has an issue at least for me. Until Wyze cleans up whatever special way they are using NTP in recent firmware, I am just excluding my cameras from the NTP redirect. Hopefully this helps you and others.
I excluded my cams from the port forward rule for the time being. I haven’t fully tested with the rule enabled after the upgrade but from the little I did I don’t think they will consistently work. I am guessing the newer firmware is expecting something specific from the NTP requests made at startup that it isn’t getting with the rule enabled and the cams then get stuck in a startup loop. When I have some time I may run some packet traces on cam startup to see what they are doing when the rule is disabled and enabled to compare. I am guessing Wyze doesn’t like getting the same time information from multiple server requests it is making. From OPNsense Insight, I can see these were the hosts the cams wanted for NTP when I was testing: 126.96.36.199, 188.8.131.52, 184.108.40.206, 220.127.116.11, 18.104.22.168 & 22.214.171.124.
Good sleuthing. I’ll see if I can’t get a mirror port and Wireshark going. NTP is such a “dumb” protocol to have it break an update. This will be one of those bugs that the more info one can present up front the better as support will likely glaze over.
Edit: pulling those IPs, those appear to be pool.ntp.org participants. I’ll see what it’s call in DNS if it’s anything unique to find NTP.
So what I have surmised… the cameras try to connect to a hard coded list of NTP servers. If they can’t reach those servers, they go into a reboot loop. If you then allow them to reach their hard coded NTP servers, they boot up just fine. If you then redirect NTP to your own server and reboot the camera after a successful upgrade, they go into a reboot loop. It seems that the firmware upgrades are working fine, but this NTP issue puts the camera into a reboot loop as you aptly noticed giving the illusion that the firmware upgrade failed.
Glad that resolved the issue for you as well. Interesting that it isn’t performing a DNS lookup. Hopefully Wyze reviews these forums posts and decides to re-evaluate their code and assumptions for NTP on bootup like it was in previous firmware.
I have a ticket open with them. I sent them our findings along with a link to this thread. So, they should know. Thanks for chiming in on this.
As an aside, my gut is that they are setting up some sort of tunnel and need good time to make things key properly. My assumption is they are making a tunnel before DNS resolution is alive during the boot process. I have seen similar in other embedded systems and it’s usually a coding oversight as they are using default Busybox or the like and simple tweaks bring up DNS first.