But surely the receiving server could validate that signature by verifying the existence of the received activity (by asking the origin server for the object referenced and validating the signature).
If like objects are distributed in URL form, this is already how it works. The extra load wouldn’t be fun, for sure, but the lack of an embedded signature makes it very easy to falsify anything on the Fediverse.