I tried SD versus HD and it helped a little. The bigger impact was when I changed which in-home wifi network I was on - it made a significant difference. Much happier with the quality of the feed but still not as good as the in-app. Will keep trying to optimize; performance in my case appears to be directly related to my wifi network and not the RTSP firmware.
Note to self - spend the money on better wifi routers in the future!