Have you tried formatting your SD card in your computer before putting the RTSP Files on it? It could possibly be something with the card (s). There is an industry standard program recommended for use in formatting SD cards available Here. Be sure to use the full (deep) format option which will overwrite the card and either fix or let you know if there are bad spots on the card.
However, it really sounds like you are not holding the setup button fully down for the full length of time. It is just a little plastic’ lever’ that has to make contact with and depress the actual button inside the cam. It has a habit of being both too sensitive when you don’t want to hit it and also sometimes difficult to hit when you do want to.
You didn’t say if you are using a pan cam or not but it can be very difficult because of the rotation of the base.
This procedure should work if you follow it correctly. If it doesn’t, I would suggest you Contact Wyze Support. Be sure to detail what you have already done and post back to let us know what happens.