user-argent: *# disallow all crawlers access to certain pages.
Disallow:/httpdocs
Disallow:/index
Allow:/
Allow:/myfile.html