Wyze AI 10 Day Grade Card

Wyze AI
1

Purpose

Summary

With the most recent Wyze AI and Firmware updates having been released, I wanted to put a quantitative effectiveness value on the Wyze AI recognition and Push Notification features operating in the wild (not in forced active testing). I originally wanted to measure:

  1. Ratio of properly tagged AI Events vs incorrectly tagged AI Events.
  2. Ratio of tagged AI Events vs untagged “missed” events.
  3. Ratio of AI Event Push Notifications received vs not received.
  4. Average Latency of Push Notifications from Event triggers.

However, my second goal would require the scrubbing of over 2600 hours of video and I neither had the time nor the AI software to complete this lofty goal. I also abandoned the fourth goal shortly into my log. Because video Events can be up to 5 minutes, and because the AI event may occur anywhere within that 5 minute timeline, I found that the AI push notifications could be delayed my many minutes because the AI Bot doesn’t pull the AI notification trigger until it actually identifies the AI object, which could be at the end of the video. With the notification using the beginning of the motion activated video in the notification text timestamp rather than the exact AI object recognition time, there was no way with any certainty that I could pinpoint the AI object identification time on the server to compare it with the time the notification was received.

Equipment

Summary

I am using 12 cams (10 V3, 1 VDBv1, 1 WCPv1) all subscribed to CamPlus, all with Smart AI recording and notifications turned on, all in operation for over a year, all with the latest Firmware installed. I am using MacroDroid running in the background to log every push notification received from Wyze.

Daily Results (10 Days):

Summary

Day 1: 6/28/22

39 AI Event Videos recorded
35 AI Event Notifications received (90%)
18 AI Event Videos correctly tagged (46%)
21 AI Event Videos incorrectly tagged (54%)

Day 2: 6/29/22

62 AI Event Videos recorded
57 AI Event Notifications received (92%)
47 AI Event Videos correctly tagged (76%)
15 AI Event Videos incorrectly tagged (24%)

Day 3: 6/30/22

54 AI Event Videos recorded
:warning: 1 AI Event Video Upload Error
50 AI Event Notifications received (93%)
40 AI Event Videos correctly tagged (75%)
13 AI Event Videos incorrectly tagged (25%)

Day 4: 7/1/22

50 AI Event Videos recorded
42 AI Event Notifications received (82%)
:warning: 1 Notification received with NO AI Event Video present
34 AI Event Videos correctly tagged (68%)
16 AI Event Videos incorrectly tagged (32%)

Day 5: 7/2/22

71 AI Event Videos recorded
73 AI Event Notifications received (96%)
:warning: 5 Notifications received with NO AI Event Video
:warning: 3 AI Event Videos without notifications (68/71)
65 AI Event Videos correctly tagged (92%)
6 AI Event Videos incorrectly tagged (8%)

Day 6: 7/3/22

126 AI Event Videos recorded
72 AI Event Notifications received (57%)
:warning: Complete notification blackout from 2:30 AM ET to 4:59 AM ET
38 AI Event Videos correctly tagged (30%)
88 AI Event Videos incorrectly tagged (70%)

Day 7: 7/4/22

44 AI Event Videos recorded
45 AI Event Notifications received (93%)
:warning: 4 Notifications received with NO AI Event Video
:warning: 3 AI Event Videos without notifications (41/44)
30 AI Event Videos correctly tagged (68%)
14 AI Event Videos incorrectly tagged (32%)

Day 8: 7/5/22

43 AI Event Videos recorded
45 AI Event Notifications received (98%)
:warning: 3 Notifications received with NO AI Event Video
:warning: 1 AI Event Video without notification (42/43)
26 AI Event Videos correctly tagged (60%)
17 AI Event Videos incorrectly tagged (40%)

Day 9: 7/6/22

70 AI Event Videos recorded
63 AI Event Notifications received (86%)
:warning: 1 Notification received with NO AI Event Video
:warning: 8 AI Event Videos without notification (62/70)
53 AI Event Videos correctly tagged (76%)
17 AI Event Videos incorrectly tagged (24%)

Day 10: 7/7/22

84 AI Event Videos recorded
86 AI Event Notifications received (99%)
:warning: 3 Notifications received with NO AI Event Video
:warning: 1 AI Event Video without notification (83/84)
36 AI Event Videos correctly tagged (43%)
48 AI Event Videos incorrectly tagged (57%)

10 Day Summary:

642 AI Event Videos recorded
552 AI Event Notifications received (86%)
387 AI Event Videos correctly tagged (60%)
255 AI Event Videos incorrectly tagged (40%)
:warning: 1 AI Event Video Upload Error
:warning: 17 Notifications received with NO AI Event Videos

Passing Grade?

6 Likes
2

For a security camera–hardly. For just a notification of interest–maybe.

1 Like
3

Not amazing numbers, but better than I had expected based around the amount of complaints.

Would be interesting to do this exact same test in a few months from now, or after the next ai update and compare results.

1 Like
4

I haven’t done an official test and actually got numbers, but it does seem like the person detection accuracy of cam plus lite has gone down a tad in the last few months.

It used to be 99% accurate, but lately it seems to be more like 70-80%

2 Likes
5

I agree. I didn’t compile the exact numbers, but the clearly evident trend I found was that the overwhelming majority of incorrectly tagged AI events were objects tagged as “Person”. I would be confident speculating over 90%.

This trend reveals, in the case of my V3 cams, that Wyze needs to focus efforts on training the AI to ignore: bugs and spider webs\cobwebs in NV IR mode, fast moving distant cars, waving plant growth, and moving sunbeams \ shadows. Most of these came in as Person AI, but there were some Pets sprinkled in there.

2 Likes
6

Interesting stats. You should do this every 2 months or so. Technically they update the AI every month, but it isn’t the same day every month, so if you did it exactly every 30 days, there would be some times where you would be retesting despite the monthly update not happening yet. Instead, if you do this every 2 months, then you are basically guaranteed to be testing it after there has been at least one AI update since your last update (and possibly 2 or or one happening in the middle of your testing).

Either way, it could be interesting to see if things change, though I am assuming you will make some modifications over time and those changes could reflect some of the differences…overall in the long run AI updates and app improvements should generally demonstrate some progress. Could be interesting data implications, even if not a lab setting or conclusive, still interesting.

1 Like
7

My primary motivation for logging this was because I learned from @WyzeJimmy that they had removed the 6 second push notification restriction wherein users with motion notifications off and AI notifications on would not receive any notifications if the AI Bot took longer than 6 seconds to process a video and tag an AI object. Because CP motion activated videos can be up to 5 minutes with the AI object coming into the field late, many notifications were never sent.

Probably not going to happen unless I experience another drop in effectiveness. This process took far too much time scrubbing 642 event videos. I was also not able to mute my notifications or modify any cam settings during this time. It felt a bit restrictive.

IMO, rather than rely on users to scrub video and share AI mistags, which I have admittedly stopped doing because of the sheer volume of AI videos produced (I had 175 in one day to scrub), it would be beneficial for Wyze to offer an opt in participation program on user accounts wherein access to all cloud videos is granted to Wyze so that a Wyze team member can do all the scrubbing thru an automated process. It might even be beneficial to run those videos thru other licensed AI software to get an idea of competitive comparison and analytics.

3 Likes
8

Nice! I totally missed this. Comports with my spidey * general sense. :+1:

  * Whoops. Not a comics guy. :sheep: -ish

1 Like