Moonrise2473 , 5 hours ago (edited 4 hours ago)

A search engine can’t pay a website for having the honor of bringing them visits and ad views.

Fuck reddit, get delisted, no problem.

Weird that google is ignoring their robots.txt though.

Even if they pay them for being able to say that glue is perfect on pizza, having

<span style="color:#323232;">User-agent: *
</span><span style="color:#323232;">Disallow: /
</span>

should block googlebot too. That means google programmed an exception on googlebot to ignore robots.txt on that domain and that shouldn’t be done. What’s the purpose of that file then?

Because robots.txt is completely based on honor (there’s no need to pretend being another bot, could just ignore it), should be

<span style="color:#323232;">User-agent: Googlebot
</span><span style="color:#323232;">Disallow:
</span><span style="color:#323232;">User-agent: *
</span><span style="color:#323232;">Disallow: /
</span>