User-agent: * Disallow: /cgi-bin Disallow: /cgi-sys Sitemap: https://www.file.net/google_sitemap.gz User-Agent: AhrefsBot Disallow: / # Crawl-Delay: 20 User-agent: MJ12bot Disallow: / User-agent: Mediatoolkit Disallow: / User-agent: Ezooms Disallow: / User-agent: linkdexbot Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: proximic Disallow: / User-agent: MegaIndex.ru Disallow: / User-agent: BLEXBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: ExtLinksBot Disallow: / User-agent: weborama-fetcher Disallow: / User-agent: expo9 Disallow: / User-agent: spbot Disallow: / User-agent: CCBot Disallow: / User-agent: Mangoway Disallow: / User-agent: Wotbox Disallow: / User-agent: GPTBot Disallow: /