The robots.txt file indicates the pages or folders that you do not want to be crawled or indexed by search engines. It actually informs the spiders about certain site pages which should not be accessed.
The robots.txt file should be placed in the root directory on the assumption that the spiders will know right away which URLs should not be indexed.
Take a look at this example:
User-agent: *
Disallow: /images/
The User-agent line contains the * character which stands for the robots, while the Disallow line contains the file or directory that shouldn't be crawled by robots. In the robots.txt file above, you are informing robots (*) that the files in the /images/ directory are off-limits.