User-agent: Googlebot Allow: / Disallow: /WebSite/ScriptResource.axd Disallow: /WebSite/WebResource.axd Disallow: /ScriptResource.axd Disallow: /WebResource.axd User-agent: bingbot Allow: / Disallow: /WebSite/ScriptResource.axd Disallow: /WebSite/WebResource.axd Disallow: /ScriptResource.axd Disallow: /WebResource.axd Sitemap: http://iatse.com/site_map.aspx User-agent: DTS Agent Disallow: / User-agent: DigExt Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Zeus Disallow: / User-agent: Niki-Bot Disallow: / User-agent: dotbot Disallow: / User-agent: SMTBot Disallow: / User-agent: AboutUsBot Disallow: / User-agent: msnbot Disallow: / User-agent: Slurp Disallow: / User-agent: meanpathbot Disallow: / User-agent: Gigabot Disallow: / User-agent: BLEXBot Disallow: / User-agent: spbot Disallow: / User-agent: SurveyBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: Icarus6j Disallow: / User-agent: Icarus6j - (contact: phil@icarus6.com) Disallow: / User-agent: oBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: XoviBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: BacklinkCrawler Disallow: / User-agent: 008 Disallow: / User-agent: Wire Disallow: / User-agent: Orthogaffe Disallow: / # Crawlers that are kind enough to obey, but which we'd rather not have # unless they're feeding search engines. User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / # Some bots are known to be trouble, particularly those designed to copy # entire sites. Please obey robots.txt. User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / # # Sorry, wget in its recursive mode is a frequent problem. # Please read the man page and use it properly; there is a # --wait option you can use to set the delay between hits, # for instance. # User-agent: wget Disallow: / # # The 'grub' distributed client has been *very* poorly behaved. # User-agent: grub-client Disallow: / # # Doesn't follow robots.txt anyway, but... # User-agent: k2spider Disallow: / # # Hits many times per second, not acceptable # http://www.nameprotect.com/botinfo.html User-agent: NPBot Disallow: / # A capture bot, downloads gazillions of pages with no public benefit # http://www.webreaper.net/ User-agent: WebReaper Disallow: / User-agent: Purebot/1.1 Disallow: / User-agent: Jakarta Commons-HttpClient* Disallow: / User-agent: Jakarta Commons-HttpClient/2.0.2 Disallow: / User-agent: ICCrawler - ICjobs Disallow: / User-agent: Test Certificate Info Disallow: / # Proprietary German backlinks service. User-agent: SEOkicks-Robot Disallow: / # Entireweb User-agent: Speedy Disallow: / # Poorly behaved bot User-agent: Ezooms Disallow: / # Russian image search engine User-agent: YandexImages Disallow: / # Yoydao User-agent: YodaoBot Disallow: / # NerdByNature.Net User-agent: NerdByNature.Bot Disallow: / # Discovery Engine User-agent: discobot Disallow: / # No idea. User-agent: knowaboutBot Disallow:/ # Gnip User-agent: UnwindFetchor Disallow: / User-agent: Java/1.6.0_22 Disallow: / User-agent: Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze) Disallow: / User-agent: FAST Enterprise Crawler/6.7.0 (crawler@fast.no) Disallow: / User-agent: FAST Enterprise Crawler/5.3.4 (crawler@fast.no) Disallow: / User-agent: Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.org/bot.html) Disallow: / User-agent: Node/simplecrawler 0.5.2 (https://github.com/cgiffard/node-simplecrawler.git) Disallow: / User-agent: panscient.com Disallow: / User-agent: Y!J-ASR/0.1 crawler Disallow: / User-agent: semanticbot Disallow: / User-agent: Aboundexbot Disallow: / # Foreign-language bot User-agent: sogou spider Disallow: / User-agent: Sogou Pic Agent Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98) Disallow: / User-agent: Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: ClaudeBot Disallow: / User-agent: ClaudeBot/1.0 Disallow: /