User-agent: ia_archiver Disallow: / User-agent: Page2RSS Disallow: / User-agent: Yandex Disallow: / User-agent: Sogou-Test-Spider Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot Crawl-delay: 10 User-agent: Site 24 X 7 Disallow: / User-agent: aiHitBot Disallow: / User-agent: panscient.com Disallow: / User-agent: SurveyBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: dotbot Disallow: / User-agent: MLBot Disallow: / User-agent: NaverBot Disallow: /