User-agent: * Disallow: /cgi-bin/ Disallow: /*.txt$ Disallow: /*.cgi$ User-Agent: Googlebot Disallow: /*.cgi$ User-agent: wget Disallow: / User-Agent: OmniExplorer_Bot Disallow: /