User-agent: MJ12bot Disallow: / User-agent: YandexBot Disallow: / User-agent: Yandex Disallow: / User-agent: AhrefsBot Disallow: / User-agent: spbot Disallow: / User-agent: BLEXBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MauiBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: barkrowler Disallow: / User-agent: Barkrowler Disallow: / User-agent: Barkrowler/0.9 Disallow: / User-agent: PetalBot Disallow: / User-agent: serpstatbot Disallow: / User-agent: serpstatbot/2.1 Disallow: / User-agent: Bytespider Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: dotbot Disallow: / User-agent: FemtosearchBot/1.0 Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: DataForSeoBot/1.0 Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: Amazonbot/0.1 Disallow: / User-agent: Amazonbot Disallow: / User-agent: my-tiny-bot Disallow: / User-agent: Claude-Web Disallow: / User-agent: GPTBot Disallow: / User-agent: CCbot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: SeekportBot Disallow: / User-agent: FriendlyCrawler/1.0 Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) Disallow: / User-agent: facebookexternalhit/1.1 Disallow: / User-agent: facebookcatalog/1.0 Disallow: / User-agent: User-agent: VelenPublicWebCrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: BDCbot/1.0 Disallow: / User-agent: ClaudeBot Disallow: / User-agent: DataForSeoBot/1.0 Disallow: / User-agent: Amazonbot/0.1 Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: FemtosearchBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-SearchBot/1.0 Disallow: / User-Agent: * Disallow: /login Disallow: /sign_up #Disallow: /rss #Disallow: /search Disallow: /wordcloud Disallow: /wordcloud2 Disallow: /h/*/new Disallow: /h/*/wordcloud #Disallow: /h/*/search #Disallow: /h/*/rss Disallow: /domain/*/new Disallow: /domain/*/wordcloud #Disallow: /domain/*/search Disallow: /user/*/new Disallow: /user/*/comments Disallow: /user/*/submitted Disallow: /user/*/wordcloud #Disallow: /user/*/search Disallow: /comment_share/* #this works, but the /search rule above actually works first #Disallow: /*?q= #no idea if this does the same thing or is more acurate #Disallow: /*?q=* #Disallow: /*?after= #Disallow: /*?before= #Disallow all query strings - might not be the best solution #Disallow: /*?*