Googles hidden files

[MYMOMMAGOTBEEF]

Just snooping around and found google's robot file. http://google.com/robots.txt

[mjmorrell]

Wow that is where every website's robot file has to be if they have it :l

[Heff]

Just snooping around and found google's robot file. http://google.com/robots.txt

You are an idiot, everyone has a robots file to tell what the robots should crawl