It looks like it goes away (I went into the settings for one contact, where there's a toggle to switch back to SMS/MMS).
Unfortunately, I can't figure out how to change it back to RCS for that conversation now.
Also, on a more personal note: the baby blue background is ugly AF. Like actually just heinous. I'd rather it wasn't there, you get a perfectly good indicator RIGHT ABOVE THE TEXT INPUT FIELD what it's about to send.