There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

maxenmajs ,
@maxenmajs@lemmy.world avatar

Alright then. The 3rd party app drama already pushed me here. I really won’t go back for anything if I’m not allowed to search for Reddit anymore.

Brewchin ,

Parts of the Internet now only searchable on specific sites now? What next - charging a monthly subscription to use Google?

This needs to be regulated before the Internet becomes like streaming TV.

tal , (edited )
@tal@lemmy.today avatar

Robots.txt has been around for a long time, and all the major search engines will honor it. Not having a full index of the Web is the norm.

That isn’t to say that the practice of signing agreements isn’t potentially a concern. Not sure that I like the idea of search engines paying sites money to degrade search results of competitors.

woelkchen ,
@woelkchen@lemmy.world avatar

test site:reddit.com works fine from DDG for me.

tal , (edited )
@tal@lemmy.today avatar

Older results will still show up, but these search engines are no longer able to “crawl” Reddit, meaning that Google is the only search engine that will turn up results from Reddit going forward.

Robots.txt lets you ask specific user-agents not to index the site. My guess is that that’s how they restricted it. I don’t know how those changes are reflected in existing indexed pages – don’t know if there’s any standard there – but it’ll stop crawlers from downloading new pages.

Try searching for new posts, see how DDG/Bing compares to Google.

EDIT: Yeah. They’ve got a sitewide ban for all crawlers. That’d normally block Google’s bot too, but I bet that they have some offline agreement to have it ignore the thing, operate out-of-spec.

www.reddit.com/robots.txt


<span style="color:#323232;">User-agent: *
</span><span style="color:#323232;">
</span><span style="color:#323232;">Disallow: / 
</span>

Here’s a snapshot on archive.org’s Wayback Machine from April 30 of this year. Very different:

web.archive.org/web/20240430000731/…/robots.txt

squidspinachfootball ,

iirc, isn’t robots.txt more of a gentlemen’s agreement? I vaguely recall bots being able to crawl a site regardless, it’s just that most devs respect robots.txt and don’t. Could be wrong though, happy to be corrected.

tal , (edited )
@tal@lemmy.today avatar

Sure, you can write software that violates the spec. But I mean, that’d be true for anything that Reddit can do on their end. Even if they block responses to what they think are bots, software can always try hard to impersonate users and scrape websites. You could go through a VPN, pretend to be a browser being linked to a page.

But major search engines will follow the spec with their crawlers.

EDIT: RFC 9309, Robots Exclusion Protocol

datatracker.ietf.org/doc/html/rfc9309

If no matching group exists, crawlers MUST obey the group with a user-agent line with the “*” value, if present.

To evaluate if access to a URI is allowed, a crawler MUST match the paths in “allow” and “disallow” rules against the URI.

EDIT2: Even if, amusingly, Google apparently isn’t for this particular case with GoogleBot, given the way that they’re signing agreements. They’ll honor it for sites that they haven’t signed agreements with, though.

EDIT3: Actually, on second thought, GoogleBot may be honoring it too. GoogleBot may not be crawling Reddit anymore. They may have some “direct pipe” that passes comments to Google that bypasses Google’s scraper. Less load on both their systems, and lets Google get real-time index updates without having to hammer the hell out of Reddit’s backend to see if things have changed. Like, think of how Twitter’s search engine is especially useful because it has full-text search through comments and immediately updates the index when someone comments.

eager_eagle ,
@eager_eagle@lemmy.world avatar 

<span style="color:#323232;">User-Agent: bender
</span><span style="color:#323232;">Disallow: /my_shiny_metal_ass
</span>
MyOpinion ,

These two shitty companies deserve each other.

BearOfaTime ,

Excellent! Now I won’t get reddit results and then have to filter them out!

MehBlah ,

Sounds great to me. With reddit gone maybe we can start to find what we are looking for without having to go sort through reddit.

tal ,
@tal@lemmy.today avatar

Kagi has a “search lens” specifically to search the Threadiverse. Like, they track lemmy/kbin/etc sites and you can specifically include them in their own results section, and can also have “!threadiverse” or whatever you want specifically search that.

They do the same for Usenet.

I suppose, given this new robots.txt Reddit development, that they’ll probably never have a Reddit lens, though.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •