# for scanning robots # allow all to see all User-agent: * Disallow: /cgi-bin/ Disallow: /scripts/ Disallow: /input/ Disallow: /output/