Hello everybody,
I'm just wondering,what's actually a robot.txt?What is the use of it?To what i understand,robot.txt is a file that contenting instructions to SE spiders where to crawl throughout the site.Am i correct?
What's the importance of it actually?Will it affect any of the judgement from SE?And,how can one set up a robot.txt file?
I have a site without any robot.txt.Do you think i should build one?
Thanz in advance for the input :)
Lowkei