I think the main problem is not really the number of people that are due to leave.
Most people tend to see and experience the other numbers you mentioned, i.e. 81 Mio vs 84,5 Mio.
How much of this increase can be attributed to organic population growth? How much of it is due to welcoming refugees?
54k really is ridiculously low, but I think an increasing amount of people do not know or care about this number. They just feel their lifes getting harder (not enough housing, food prices, stressed health systems…) and helplessly point their fingers on the seemingly easiest solution.