I haven’t formally tested it, but this appears to be possible in the Android app (and I assume your iOS app, as well). Hopefully an iOS user will chime in with a definitive answer.
This would appear to be the expected behavior, according to a Support article:
If there are no device(s) or service(s) being shared with the user, they will automatically be removed from the list of shared users.
As I’m still writing this, I see that @carverofchoice has already answered! (No surprise there.)
Eh…except that shared users still have limited control over devices.
Anyway, welcome to the Forum, @sllankford!