You’ll get a lot of contradictory answers with this question because of two major issues.
There is more than one way to make your scale number go down.
Your scale number going down can be for multiple reasons.
For example, dropping a bunch of body fat is a way of posing weight, but it does not look any different on the scale than losing muscle mass or losing a leg. You can have more healthy recomposition where you drop a bunch of fat slowly over time and gain some muscle but overall lose absolutely no weight on the scale, and you can also gain weight without changing fat but be in a better position.
So what would you aim for? It depends on your goals. Do you want to be jacked? Maybe you have early signs of type 2 diabetes and want to stop it there. Or maybe you just really want to get rid of your skin issues like acne and dermititis.
Nobody benefits from being insulin resistant. That is the state that pushes you towards weight gain, diabetes, heart disease, and many other issues including dementia. Fixing that is a central goal for a lot of people and it actually helps with most other health related goals. If I were starting somewhere that is where I would probably try to start.
That said, if you have very little muscle that may be better to work on.