I have done some basic testing, and the speed of the USB stick you use does make a noticeable difference on the boot time of whatever you install on it.
If I recall correctly, A low speed USB 2.0 stick took around 30-60 seconds to load (either time to login screen or time to reach a blinking cursor for something like an arch install disk). If this is something for occasional use, even this works perfectly fine.
Slightly faster USB 3 sticks in the 100MB/s range can be had for only around $5-15 USD and work significantly better, maybe 7-15 seconds. These usually have assymetric read/write speeds, such as 100MB/s read and 20MB/s write, but for a boot disk the read speed is the primary factor.
Some high end flash drives can reach 500-1000MB/s and would load in only a few seconds. A high speed 256GB stick might cost $25-50, and a 1TB stick maybe $75-150.
An NVMe enclosure might cost $20-30 for a decent quality 1GB/s USB 3 enclosure, or $80-100 for a thunderbolt enclosure in the 3GB/s range so long as your hardware supports it, plus another $50-100 for a 1TB NVMe drive itself. This would of course be the fastest, but it is also bulkier than a simple flash drive, and I think you are at the point of diminishing returns in terms of performance to cost.
I would say up to you on what you are willing to spend, how often you realistically intend to use it, and how much you care about the extra couple seconds. For me, I don’t use boot disks all that often, so an ordinary 100MB/s USB 3 stick is fine for my needs even if I have faster options available.