This is only true specifically when you’re thinking about it as a great source can’t have its credibility rating lowered. A not great factual source can get a high credibility rating if it’s deemed centrist enough which again is arbitrary based on the (effectively) 1 guys personal opinion.
The impact either way is slight. I’m sure you could find a few edge cases you could make an argument about because no methodology is perfect, but each outlier represents a vanishingly small (~0.01%) amount of their content. When you look at rigorous research on the MBFC dataset though, the effect just isn’t really there. Here’s another study that concludes that the agreement between bias-monitoring organizations is so high that it doesn’t matter which one you use. I’ve looked and I can’t find research that finds serious bias or methodological problems. Looking back at the paper I posted in my last comment, consensus across thousands of news organizations is just way too high to be explainable by chance. If it was truly arbitrary as people often argue, MBFC would be an outlier. If all the methodologies were bad, the results would be all over the map because there are many more ways to make a bad methodology than a good one. What the research says is that if one methodology is better than the others, it isn’t much better.
Again, I think you make a really good argument for why MBFC and sites like it shouldn’t be used in an extreme, heavy-handed way. But it matters if it has enough precision for our purposes. Like, if I’m making bread, I don’t need a scale that measures in thousandths of a gram. A gram scale is fine. I could still churn out a top-shelf loaf with a scale that measures in 10-gram units. This bot is purely informational. People are reacting like it’s a moderation change but it isn’t – MBFC continues to be one resource among many that mods use to make decisions. Many react as though MBFC declares a source either THE BEST or THE WORST (I think a lot of those folks aren’t super great with nuance) but what it mostly does is say ‘this source is fine but there’s additional info or context worth considering.’ Critics often get bent out of shape about the ranking but almost universally neglect the fact that, if you click that link, there’s a huge report on each source that provides detailed info about their ownership history, funding model, publishing history, biases, and the press freedom of the country they’re in. Almost every time, there are reasonable explanations for the rankings in the report. I have not once ever seen someone say, like, ‘MBFC says that this is owned by John Q. Newspaperman but it’s actually owned by the Syrian government,’ or ‘they claim that they had a scandal with fabricated news but that never happened’. Is there a compelling reason why we’re worse off knowing that information? If you look at the actual reports for Breitbart and the Kyiv Independent, is there anything in there that we’re better off not knowing?