It’s fairly easy to add a robots.txt file to your public anonymous MOSS site. Just open up your root site in SharePoint Designer. Right-click the top folder with your site URL and select New – SharePoint Content. Click the Page tab, select Text File and click OK.
Add the robots.txt file content and save it as “robots.txt”. That’s it!
Make sure the file is accessible to search engines by going to your site URL with a “/robots.txt” at the end. You should see the contents of your robots.txt file in the browser.