Outdoor cam loses connection after power outage - and it's my fault

[rbhateja]

One of my outdoor cameras totally forgot everything. Got renamed to something generic like “Outdoor cam” on its own. And could no longer connect. My WiFi is on UPS, so that wasn’t the problem. But it could have been a power outage (base station was flashing blue). After a power cycle, the base station came back solid blue. But all the cameras lost their connection and had to be manually re-synced – a major pain since they’re not in the most accessible locations but also because what if I wasn’t home?
Here’s the response I got from support (who I’ve renamed to Support), which suggests that it’s not Wyze’s fault, it’s mine. And the “No Bugs are being reported” and “You should pay more attention to your base station.”
Hilarious! How cool is that?

(11:27:39 AM) Support: Yes indeed. But as for the cameras being on offline state, has to be resynced, to minimize this issue, you have to make sure the wifi stays connected. Otherwise, manual resync would be the only option.
(11:29:06 AM) Me: OK. That’s not very practical. I have my WiFi on a UPS. So I don’t think the WiFi going down was the problem at all.
(11:31:29 AM) Support: I understand, But if the outdoor cameras are offline, a remote resync is not feasible.
(11:32:29 AM) Support: Hopefully, we can add it somehow in the future updates.
(11:34:02 AM) Me: OK. But the question is why is a resync even necessary. If the cameras have battery. And the WiFi never went down?
The only possibility is that the cameras don’t recover after a power outage of the base unit. Or there’s a bug in the firmware.
(11:38:08 AM) Support: No Bugs are being reported so far. camera’s sometimes cannot recover if the base was been disconnected from the wifi.
(11:40:31 AM) Support: You should pay more attention to your base station, and make sure it’s showing solid blue light all the time because your camera relies on the connectivity to from your base.

Customer service at its best!

[carverofchoice]

That is frustrating. Sorry to hear of the problem with your WCOs.

@rbhateja

the question is why is a resync even necessary. If the cameras have battery. And the WiFi never went down?
The only possibility is that the cameras don’t recover after a power outage of the base unit. Or there’s a bug in the firmware.

It probably had something to do with the firmware update to the base station. New firmware was released on 1/24/24.

While a bug in the firmware IS a distinct possibility, that is not the only possible cause. Electronics are fairly complex with multiple variables that can affect them. Parts also wear out over time. I am not saying the following is what happened, just pointing out that there are other possible considerations: it’s possible, for example, that even if the firmware is flawless, that a sector of the local memory that the firmware is installed on has a worn out sector on it, or maybe that piece of it didn’t get encoded properly for any number of reasons. various degrees of electrical or RF interference, physical jostling of some kind or any number of other things while it is downloading or installing the firmware, or like I said, a sector of the memory chip could be wearing out (though unlikely). There are so many things that can contribute to it. It is hard to know the cause. I know this firmware did go through beta testing, and I installed it myself with no problems, so it’s definitely not happening to everyone. It would be nice to figure out why it reacted differently in this case though and not to everyone. I wonder what the difference could be. My bet is that ultimately the firmware update process was corrupted in some way and now somehow requires the base station to be setup again.

@rbhateja

(11:40:31 AM) Support: You should pay more attention to your base station, and make sure it’s showing solid blue light all the time because your camera relies on the connectivity to from your base.

Support should have left this out. It’s not relevant to this particular issue.

@rbhateja

(11:38:08 AM) Support: No Bugs are being reported so far. camera’s sometimes cannot recover if the base was been disconnected from the wifi.

This isn’t intended to be offensive. It is not implying that it is your fault or that your issue isn’t being believed. It is explaining why this isn’t a known issue. It’s not widespread, it wasn’t found during internal testing, it wasn’t reported by anyone during beta testing, etc. As far as I’m aware, this is the first report of this firmware update not going through smoothly. This indicates there is some other rare variable involved that doesn’t apply to everyone else, or there would’ve been lots more reports of this happening. Maybe there is a bad sector in the firmware memory chip? Or a short in a connection going to it? IDK. That doesn’t mean it’s your fault either. Electronics do wear out among things. That is why electronics usually have a short warranty period. They aren’t expected to last forever, but some do seem to sometimes. There is generally a normal curve with some exceeding the standard deviation and others not. Fault isn’t necessarily the issue here. The reason they are cautioning you to be careful about the cameras not recovering is because the WCO’s have an anti-theft measure in them. If they are deleted from the app while they are offline, they are permanently bricked and can never be setup again. So, they are trying to caution you here that whatever else you do, DO NOT delete them from the app. If you need to go through setup again to pair them back to the base, that’s fine, go through setup, but DO NOT delete them from the app first while they are offline, or you will permanently brick them. Same with the Base Station. Never delete them while they are offline, even if you are just trying to fix/restore them to functionality.

Basically, nobody is blaming you for it. The firmware update failed for some reason and the device reset to factory settings. That is frustrating to be sure! I don’t think anyone disagrees with that. Whatever the cause, I don’t think anyone, including support, thinks it is your “fault.” But it will probably require going through setup again.

That is basically the summary of what the rep is saying…need to go through setup again. I don’t think anyone can answer WHY it happened, and I am not sure the WHY would change anything even if we could know what the WHY is.

I have actually had to do this with my base station and WCO in the past, so I do know the frustration. I didn’t have an issue with this last firmware update, but I have had an issue in the past that forced me to go through setup again.

I hope you’re able to get your stuff up and running again. Just be sure you don’t delete the devices from your app before you get them up and running again.

[rbhateja]

Thanks for the thoughtful, detailed response. I appreciate it.

The summary seems to be that this could be a hardware problem, not necessarily firmware, so there’s nothing that can be done and it’s no one’s fault. May be I’m oversimplifying but that’s kind of a rough gist.

Here we disagree. There are always solutions. There could be a BIST (built-in self-test) routine that is run every time the device goes through boot up to diagnose hardware problems. Many systems do this. This one apparently doesn’t. Other solutions (retry until successful) are also possible and could correct other problems.

So I maintain two things:

1. There is a solution. I’ll agree that no solution is a 100% guarantee but it could be better than the way the system behaves now.
2. The response “You should pay more attention to your base station” is not appropriate. Apparently, we agree on that.

Thanks again.

[carverofchoice]

You have some excellent thoughts and you are clearly solution and prevention oriented which is an awesome skill.

@rbhateja

The summary seems to be that this could be a hardware problem, not necessarily firmware, so there’s nothing that can be done and it’s no one’s fault. May be I’m oversimplifying but that’s kind of a rough gist.

It’s hard to know for sure, but I wanted to present a possibility along those lines. you understood my alternate example. Having said that, you could still absolutely be correct that the firmware has a bug in it. I don’t want to say that isn’t also a possibility.

@rbhateja

There are always solutions. There could be a BIST (built-in self-test) routine that is run every time the device goes through boot up to diagnose hardware problems. Many systems do this.

I don’t know whether it does or doesn’t, but it seems plausible that it does not. Regardless, I do think that self-test routines are a lot more complicated and less guaranteed than most people realize. I addressed that a little bit in another thread here:

@carverofchoice

Hi professional QA tester here. The quoted request is based on a misunderstanding of the nature and complexities of programing. Anyone with any professional programming or QA testing experience will explain that what is being sought here is absolutely guaranteed 100% utterly impossible for ANY company in our current situation with electronics and using programming to ever promise anyone.

Let me give you a quite simple example. In the following control flow example, this “simple” program consists of up to only FOUR decisions. Just 4 decisions and a loop (even with just 4 decision trees, it could’ve been made almost doubly more complicated if we expanded the decisions on the far right and each subsequent decision…but we’re keeping it as simple as possible). For this SUPER SIMPLE piece of programming/code, if you were to EXHAUSTIVELY test every possible combination of execution with a limit of only 20 cycles, then if it took the tester 5 minutes to execute each test sequence, then complete testing on this small piece of code with as few as 14 lines of code and 4 total decisions to make, it would take the testers ONE BILLION YEARS to finally test everything to be sure it was “fixed period and without exception and in all manner.” One billion years on even the absolute simplest code because even for something so simple it seems almost worthless, there are actually roughly 100 quadrillion sequences of just this stupid easy program.

If a person says, “yeah, but you can run most of the tests automatically and fast for each test case instead of a human testing each case.” Fair enough point in some cases (not always true, but we’ll pretend it is in this case). Let’s assume that each test now only takes FIVE MICRO SECONDS instead of 5 minutes. That’s a more than fair adjustment. That means complete testing on this small piece of code with as few as 14 lines of code and 4 total decisions to make, it would take automatic testing NINETEEN YEARS to finally test everything to be sure it was “fixed period and without exception and in all manner.”

Basically, if anyone is looking for a promise from Wyze (or ANY company that uses programming of any sort) that something is 100% fixed without bugs, etc…it’s NEVER, NEVER going to happen. They would have to be completely insane to ever promise anyone that. Research has proven this fairly conclusively, and that is just for the most simple of code…let alone all the other huge complexities of millions of lines of code and decisions and loops actually being used. Everything Wyze has is infinitely more complex than this smallest of tiny programs.

A promise of no bugs will never be forthcoming. It’s nearly impossible. Best we can do is get indications that most things work okay most of the time with few known critical errors reported with any frequency. It also means if a person wants more stability they should stick to production versions of firmware and software and stay away from beta like the plague. Beta is for helping to test and find the most common and critical bugs and problems which almost for sure, guaranteed, exist in some form…some of them we will find immediately, some of them might take billions of years of testing to discover, but statistically it is pretty guaranteed there are bugs.

If such guarantees are needed, then I would suggest giving away or selling all Wyze devices now because this promise is not only not EVER coming, it is absolutely impossible. Wyze can not possibly eliminate every single bug within our lifetimes. It would be hard enough if their code was only a few lines long and a single product. With everything they have and the amount of code they are using…impossible, utterly and completely impossible. It’s not their fault, it’s math. It’s normal.

This is why I say that it is possible it IS still a firmware issue (it’s hard to know). Even if this didn’t happen to 99% of other people who got the firmware update, that doesn’t mean the bug isn’t there and it might affect some people like this. It is mathematically IMPOSSIBLE to ever exhaustively test every single piece of code even when it is just a few lines long and a few small decisions. So, it absolutely could be that. It may not even be possible for anyone to ever know for sure. That is why I said:

@carverofchoice

I don’t think anyone can answer WHY it happened, and I am not sure the WHY would change anything even if we could know what the WHY is.

The end result is, regardless of WHY, the next step is to try to set it up again. Your experience will be tracked and if a pattern emerges of other reporting the same thing, then a comparison can be made between the logs and the cause can be narrowed down and explored. It is hard to do much on a single use-case without other confirmations to help figure out what happened.

If I find any similar examples I’ll be sure to pass them on with yours so Wyze can figure out the cause, resolve it and prevent it in the future.

[rbhateja]

Thank you for a well stated objective assessment.

Bottom line for me:
This lack of unreliability (and it’s certainly not the only instance I’ve run into) is very inconvenient and particularly worrisome in a security product.

[carverofchoice]

I appreciate how you made it a pleasurable discussion for both of us. You have some good thoughts and presented them in a respectful manner. Again, sorry this happened to you. I agree that it is VERY inconvenient and frustrating for sure and leave a bad impression about reliability.

Let us know if you eventually get it back up and running okay so we know things we volunteers can help recommend to others if a similar situation comes up.

[rbhateja]

A daily reboot of the base station seems to have helped considerably.

[carverofchoice]

That’s good to hear. Ideally, it shouldn’t be necessary, but I’m glad you found something that helps. Hopefully future updates can help improve this.