Being a temp worker who can be kicked out at any time would not imply you’re wanted there - so it’s kind of a given. Even if we factor residency into it, you’re looking at 3% of the population being non-Japanese.
Japan is famously xenophobic, just because you’re allowed to work there doesn’t change that. You’ve been there a decade, do the locals consider you Japanese?