Those who wish to utilize search engine optimization are most likely attempting to have their site and its content picked up by search engines, as much and as often as possible. There are exceptions to what content a site would like to appear as a result of a search engine. A site may include areas that they want search engine crawlers to stay away from in a robots.txt file that will instruct search engine 'crawlers' which areas to stay away from and is often the file these crawlers seek first.
Anyone seeking to learn SEO or in search of SEO help will first need help in figuring out what will need to go into a robots.txt file (if anything) and what might possibly detract from the other SEO techniques. SEO companies and other professionals may be able to offer SEO tips as to the proper use of the robots.txt file and results will soon be made apparent while disseminating the web analytics.