It is an objective fact that a harmful act harms someone. That one observer likes that outcome does not alter the objective moral weight of the act. Harmful acts are objectively wrong, regardless of preference.
From a basic empirical observation of the effects of harm, one can arrive at a moral system based on objective reasoning. In this way, ideology can be avoided.