Update on Investigation of 2/16/24 Security Issue

Wyze News
,
1

We have sent emails out to all affected and unaffected Wyze users from the security issue that occurred on 2/16/24.

The first email went to all unaffected users.

The second email went to users whose event thumbnails were made available to others but not tapped on.

The third email went out to users whose event thumbnails were made available to others and were tapped on.

The fourth email went out to users who had thumbnails made available to them that were not their own, but their thumbnails were not made available to others.

The following posts contain copies of these emails.

There may be a slight delay in receiving your email copy due to the large number of emails sent. If you still have not received your copy of one of these emails, please check your email spam folder.

Edit: Here is a link to the latest update from @WyzeDave

10 Likes
3

Email to all unaffected users:

Wyze Friends,

On Friday morning, we had a service outage that led to a security incident. Your account and over 99.75% of all Wyze accounts were not affected by the security event, but we wanted to make you aware of the incident and let you know what we are doing to make sure it doesn’t happen again.

The outage originated from our partner AWS and took down Wyze devices for several hours early Friday morning. If you tried to view live cameras or Events during that time, you likely weren’t able to. We’re very sorry for the frustration and confusion this caused.

As we worked to bring cameras back online, we experienced a security issue. Some users reported seeing the wrong thumbnails and Event Videos in their Events tab. We immediately removed access to the Events tab and started an investigation.

We can now confirm that as cameras were coming back online, about 13,000 Wyze users received thumbnails from cameras that were not their own and 1,504 users tapped on them. Most taps enlarged the thumbnail, but in some cases an Event Video was able to be viewed. All affected users have been notified. Your account was not one of the accounts affected.

The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.

To make sure this doesn’t happen again, we have added a new layer of verification before users are connected to Event Videos. We have also modified our system to bypass caching for checks on user-device relationships until we identify new client libraries that are thoroughly stress tested for extreme events like we experienced on Friday.

We know this is very disappointing news. It does not reflect our commitment to protect customers or mirror the other investments and actions we have taken in recent years to make security a top priority at Wyze. We built a security team, implemented multiple processes, created new dashboards, maintained a bug bounty program, and were undergoing multiple 3rd party audits and penetration testing when this event occurred.

We must do more and be better, and we will. We are so sorry for this incident and are dedicated to rebuilding your trust.

If you have questions about your account, please visit support.wyze.com.

Wyze Team

6 Likes
4

Email to users whose event thumbnails were made available to others but not tapped on.

Wyze Friends,

On Friday morning, we had a service outage that led to a security incident affecting your Wyze account.

The outage originated from our partner AWS and took down Wyze devices for several hours early Friday morning. If you tried to view live cameras or events during that time you likely weren’t able to. We’re very sorry for the frustration and confusion this caused.

As we worked to bring cameras back online, we experienced a security issue. Some users reported seeing the wrong thumbnails and Event Videos in their Events tab. We immediately removed access to the Events tab and started an investigation.

We can now confirm that as cameras were coming back online, about 13,000 Wyze users received thumbnails from cameras that were not their own and 1,504 users tapped on them. Most taps enlarged the thumbnail, but in some cases an Event Video was able to be viewed. We have identified your Wyze account as one of those affected. This means that thumbnails from your Events were visible in another Wyze account, but we have confirmed that they were not tapped and no Event Videos were viewed.

The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.

To make sure this doesn’t happen again, we have added a new layer of verification before users are connected to Event Videos. We have also modified our system to bypass caching for checks on user-device relationships until we identify new client libraries that are thoroughly stress tested for extreme events like we experienced on Friday.

We know this is very disappointing news. It does not reflect our commitment to protect customers or mirror the other investments and actions we have taken in recent years to make security a top priority at Wyze. We built a security team, implemented multiple processes, created new dashboards, maintained a bug bounty program, and were undergoing multiple 3rd party audits and penetration testing when this event occurred.

We must do more and be better, and we will. We are so sorry for this incident and are dedicated to rebuilding your trust.

If you have questions about your account, please visit support.wyze.com.

Wyze Team

4 Likes
5

Email to users whose event thumbnails were made available to others and were tapped on

Wyze Friends,

On Friday morning, we had a service outage that led to a security incident affecting your Wyze account.

The outage originated from our partner AWS and took down Wyze devices for several hours early Friday morning. If you tried to view live cameras or events during that time you likely weren’t able to. We’re very sorry for the frustration and confusion this caused.

As we worked to bring cameras back online, we experienced a security issue. Some users reported seeing the wrong thumbnails and Event Videos in their Events tab. We immediately removed access to the Events tab and started an investigation.

We can now confirm that as cameras were coming back online, about 13,000 Wyze users received thumbnails from cameras that were not their own and 1,504 users tapped on them. We’ve identified your Wyze account as one that was affected. This means that thumbnails from your Events were visible in another Wyze user’s account and that a thumbnail was tapped. Most taps enlarged the thumbnail, but in some cases it could have caused an Event Video to be viewed.

The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.

To make sure this doesn’t happen again, we have added a new layer of verification before users are connected to Event Videos. We have also modified our system to bypass caching for checks on user-device relationships until we identify new client libraries that are thoroughly stress tested for extreme events like we experienced on Friday.

We know this is very disappointing news. It does not reflect our commitment to protect customers or mirror the other investments and actions we have taken in recent years to make security a top priority at Wyze. We built a security team, implemented multiple processes, created new dashboards, maintained a bug bounty program, and were undergoing multiple 3rd party audits and penetration testing when this event occurred.

We must do more and be better, and we will. We are so sorry for this incident and are dedicated to rebuilding your trust.

If you have questions about your account, please visit support.wyze.com.

Wyze Team

4 Likes
6

Email to users who had thumbnails made available to them that were not their own, but their thumbnails were not made available to others

Wyze Friends,

On Friday morning, we had a service outage that led to a security incident affecting your Wyze account.

The outage originated from our partner AWS and took down Wyze devices for several hours early Friday morning. If you tried to view live cameras or events during that time, you likely weren’t able to. We’re very sorry for the frustration and confusion this caused.

As we worked to bring cameras back online, we experienced a security issue. Some users reported seeing the wrong thumbnails and Event Videos in their Events tab. We immediately removed access to the Events tab and started an investigation.

We can now confirm that as cameras were coming back online, about 13,000 Wyze users received thumbnails from cameras that were not their own and 1,504 users tapped on them. Most taps enlarged the thumbnail, but in some cases it could have caused an Event Video to be viewed.

We’ve identified your Wyze account as one that was able to see and tap on thumbnails that were not yours, but your thumbnails and Event Videos were not seen by anyone else.

The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.

To make sure this doesn’t happen again, we have added a new layer of verification before users are connected to Event Videos. We have also modified our system to bypass caching for checks on user-device relationships until we identify new client libraries that are thoroughly stress tested for extreme events like we experienced on Friday.

We know this is very disappointing news. It does not reflect our commitment to protect customers or mirror the other investments and actions we have taken in recent years to make security a top priority at Wyze. We built a security team, implemented multiple processes, created new dashboards, maintained a bug bounty program, and were undergoing multiple 3rd party audits and penetration testing when this event occurred.

We must do more and be better, and we will. We are so sorry for this incident and are dedicated to rebuilding your trust.

If you have questions about your account, please visit support.wyze.com.

Wyze Team

4 Likes
9

I like that this time there was an email sent to everyone identifying all the different conditions/categories of people affected (or not). I see this as a step in the right direction and something that everyone was asking for to be done differently for any future issues, so it is good to see this now. Thank you for listening to our feedback requesting you do this in the future (I certainly [strongly] requested this in the past, so thanks), and deciding to implement it.

I also appreciate that you didn’t postpone sending this and didn’t decide to wait until normal business hours to send us your results (ie: Tuesday Morning). You told us you were doing a full investigation and would tell us the results as soon as you could give us the facts. I guess your “as soon as” was totally literal because you finished the investigation and even though it’s the middle of the night on a weekend night going into a holiday, you sent out the message now anyway because it was ready and felt you shouldn’t postpone it. That’s encouraging. There may be some people not happy about middle-of-the-night notifications, but I think it was the right call for something like this (besides, who doesn’t know how to use do-not-disturb, especially for something like emails while they sleep?).

While this was serious enough that working through the holiday weekend should be a given (especially for fixing things), I am still glad that you worked through the holiday weekend and late into the night and being thorough enough with the investigation that you identified every single account affected and every account that had a wrong thumbnail/event and every account that clicked on an event. The thoroughness is appreciated and letting me/us know which group/category each of my accounts fall under is appreciated so I am not just sitting around wondering.

(Edit update: Dave just stated the investigation isn’t over yet: “This investigation is not wrapped up yet, we will continue to discuss as a leadership team and evaluate what needs to change to better protect our users” which means you sent out this update after just the mid-analysis, without making us wait even longer. I’m not sure if this means that we’ll get another update or not toward the completion. There is more information I’d love to hear, but I am glad you didn’t make us wait longer for this information that was currently known and ready.)

I think bypassing cache for checks on user-device relationship is a good move for now considering cache issues have come up twice now, though in different ways. Yes, please do stress test the caching, preferably with a 3rd party contractor before using caching again.

I think the response this time was an improvement and increasingly transparent. Certainly more than most other companies who have leaked my information. A friend of mine recently shared with me this site: https://ocrportal.hhs.gov/ocr/breach/breach_report.jsf where there is a list of recent breaches of protected health care information constantly being hacked, stolen or leaked, with tens of thousands of victims constantly happening. It was a little terrifying how often our protected health and other information is constantly being hacked/stolen and we basically never even hear anything about it. I didn’t realize they were happening ALL THE TIME, like 10s of thousands multiple times per week and nobody even mentions it, and we get few to no details at all. I appreciate that this message proactively identified every affected user and let us know and even let those unaffected know. I for sure much prefer being notified than left in the dark (as being left in the dark seems to be standard).

8 Likes
10

What is the name of the caching library in question and what kind of bug would lead to a condition where a key would return a different value depending on load? Is this bug publicly documented?

Thanks,
Jeff

4 Likes
11

I was wondering that also.

I know wyze recently started using Momento for caching of events and stuff:

But im not sure if they also use it for account stuff like that.

Definitely seems like a weird issue. It makes sense that it could happen when pushing a code change, but I don’t see how that could happen due to high demand. Unless Momentos system goes into a different state when in high demand (after AWS came back up all the queued tasks flooded the system) and the bug occurred in there.

Caching is very complicated though so anything’s possible lol.

I do appreciate the details we have been given and the email sent to everyone. That’s more than most companies will do and shows that wyze is listening to our feedback. Thank you.

4 Likes
12

I assume this is talking about new dashboards used exclusively by their security team for monitoring things, etc? And not referencing anything for us to use.

Though I would LOVE to have a fancy Wyze Dashboard (in the app or the web portal) that I could customize with all of my Wyze stuff on it! Maybe that needs to be a wishlist item! For now, I just imported most of my Wyze stuff into Home Assistant and make custom Wyze dashboards for myself there instead. :slight_smile:

3 Likes
13

Still not being responsible and blaming the AWS and a third party caching client library is weak.

Wyze is responsible for their choice of 3rd party providers, and the implementation to their software / firmware.

Each month there has been some form outage/issue lately, get your [mod edit] together Wyze :face_with_symbols_over_mouth:

6 Likes
14

Hello Jason … I was hoping to see this email in my inbox, but I did not. Not even on my provider’s email server (i.e. it was not in Spam, etc.) In fact I didn’t receive any of the 5 email permutations you describe. Shouldn’t I have received something by now? TIA

4 Likes
15

I just received mine. It may take some time before emails are received by all. Emails were sent to every account, affected and unaffected. That must be millions of emails.

3 Likes
16

Jason said on Reddit that they sent all the affected emails first, and the unaffected emails last and that those ones may take a lot of time to get out to everyone (Wyze has millions of users), so those affected should probably already have the email, and if you haven’t gotten one yet, you can probably pretty safely assume that you are not in the affected group and will be getting the email listed first up above.

I would say if you don’t have one with 24-48 hours then it could be worth asking for more info about how to tell if you were affected if you didn’t get an email.

5 Likes
17

Again? Thumbnails were sent to the wrong accounts in Sep 2023. You would have thought a critical, company-defining, you-had-one-job security lapse like that would have been fixed the first time. Or you know never allowed to happen in the first place .

Here we are again. I was such a huge wyze supporter when I first found out about the company.

The clock is ticking for wyze’s bankruptcy. It will be a case study on how a company can squander such amazing customer goodwill and word of mouth marketing.

5 Likes
18

Makes perfect sense, I should have waited until morning and then checked. In fact, I just now received the email saying that my account was not affected. @Seapup @WyzeJasonJ

4 Likes
20

I also am interested in this info and think it should be shared.

3 Likes
21

When will viewing events come back online? Are there any steps users need to take?

1 Like