This is directed mostly at those on this forum who might have some amount of insight into the software development at Wyze.
I’ve read many posts in these forums about devices (cameras mostly) that freeze or hang or crash and need to have power cycled to recover them. That has happened to me, entirely too often (not REAL often but it shouldn’t happen at all).
I understand that firmware development at Wyze is done in C and perhaps some C++. Those languages provide means to dynamically allocate and de-allocate memory (keywords “new” and “delete” in C++, “malloc” and “free” in C). You allocate some memory, do what you need to do with it, and then return it to the available memory pool or “heap”.
This is dangerous to do with limited memory resources such as those that are available in “Internet of Things” devices like cameras. You can allocate memory and then forget to de-allocate it (resulting in a “memory leak”) or you can allocate and de-allocate properly but then the “heap” can get fragmented to the point where a large-enough continguous block is no longer available. Either way, if not handled properly, a crash results. The more limited your memory resources, the sooner this is likely to happen. Cycling power re-initializes memory and all is well again… until the next time.
I’m wondering if Wyze engineers are cognizant of all of this and have taken steps to guard against it… like don’t use new/delete or malloc/free (harder to do with C++). Before I retired I consulted in software engineering at many different companies since 1980, and it’s absolutely disheartening how often degreed and experienced software engineers fail to take that kind of thing into account in their designs.
I doubt any forum users other than ex-Wyze developers have insight to Wyze software development details. I’m positive Wyze developers are aware of memory allocation/deallocation, memory leaks, threading, fragmentation, etc. during design and coding. Whether they utilize profiling tools to identify and rectify these issues in code review and testing is not something they are likely to share with us. But this would be a great question to ask during one of Wyze’s AMA (Ask Me Anything) events.
One of the non-employee Mavens (I forget which one… IEatBeans maybe?) was able to tell me that Wyze uses C/C++, and FreeRTOS in the newer designs and Linux in the older ones. So apparently there is some insight somewhere on these forums.
Of course, since you moved this to someplace where no one will see it, I guess it’s moot. Would have been good if I could have gotten some kind of notice that you had done that. When I couldn’t find it in Cameras, it occurred to me that it must have been moved or deleted, and I had to guess at where it might have gone.
In any case I thought it might be useful for those who, like me, experience these crashes that require cycling power. At least it might have helped people understand how things like that can happen.
I don’t share your confidence. I’ve consulted to many different clients over 30+ years, mostly in medical devices of all things, and it’s really disheartening how often I ran into people who should know better but are completely oblivious to things like this.
What’s an AMA event? First I’ve heard of such a beast. I’ve heard of Fix It Friday but nothing else like that.
From the Compleat Idiot’s POV (I represent this cohort well) this is pretty interesting. Wyze did a short series of talks (conversations, casual interviews) between employees in the second and third year, I think. Even though their shop talk was often beyond my full comprehension, it gave me a useful sense of their arcane world…
I have 40+ years experience in critical-systems firmware and applications architecture, design and development. Our level of experience isn’t relative to the topic, rather the opposite as the average Wyze user simply expects devices to work flawlessly vs memory management education in order to understand why a product has “issues”.
If you’d like to help expedite firmware or app bug mitigation, please submit a debug log via the Wyze app and make a post including the log ID (Log ID is critical). Any additional supporting evidence is extremely helpful, e.g., screenshots, video captures of abnormal behavior as witnessed via app, firmware/app version info, phone/tab make/model/OS version, etc. If you’re not sure where to post, just create a new topic and we’ll merge your topic to best location for Wyze visibility. When done correctly, Wyze does take notice, is eager to act, sometimes communicating back directly and is always grateful for the help. In doing so, Wyze has corrected literally every bug I’ve discovered from obscure app memory leaks, package/plugin deprecation to deployment misconfigurations.
AMA is a Wyze event where new features/prototypes are demo’d by team leaders and developers. We have the opportunity to ask the panel any questions about anything, even at the most technical level. You can find AMA announcements in the forum #news category. You may also wish to enable “Watching First Post” notifications on the #news category to be notified when Wyze posts a new topic.
I’m going to look up long-term performance data for these devices from now on…
Unfortunately performance engineering is usually one of the last areas for companies to invest in. Why bother refactoring old code when we can just tell customers to reboot? Even my Roomba (yes, the iRobot one) fails sometimes with “low memory” and I have to reboot it.
And with development happening in sprints, all the more it is difficult for any meaningful performance tests for memory leaks. In one of my earlier jobs, half the time I was trying to prove to Development that indeed there was a memory leak and it would have to be a bulletproof reproducible scenario for them to be willing to take a look.
You mention sprints, so they are really Agile? I’d be interested in knowing how long their sprints are, and how they came to that duration for each sprint. Someone said that they release a new revision of firmware and/or app software about once a month, so it sounds like their sprints are calendar-month based? That’s a little on the long side for sprints but can work well.
I have been scrum master and Agile coach on projects where the development on a user story was done in one sprint and the QA on that same story, including performance testing, was done on the next sprint. It worked reasonably well. Some say it kind of pushes the concept of delivering working code at the end of every sprint, but I say not really… you can “deliver working code” to the QA team, not necessarily to the customer or product owner. There is nothing wrong with devoting a whole user story to performance testing and memory leak detection, if that’s what the scrum team decides is the best approach.
I’d like to be a fly on the wall at a few of their daily scrums and a sprint planning meeting or two.
Cam+Floodlight crashed this morning, required power cycle to recover. Log ID 1045106.
It may have crashed as it was detecting motion, however it did not alert on the motion or show a video clip. It does appear like it was able to store a video on the SD card before it crashed.
