Forum Moderators: goodroi
I was looking at logs (disregard dates, its the same no matter the date) and saw this;;
I have my robots.txt file in main root of my site. I was checking my logs and find it says
66.249.64.52 - - [01/Aug/2005:23:17:00 -0400] "GET /robots.txt HTTP/1.0" 200 632 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
66.249.64.52 - - [01/Aug/2005:23:17:00 -0400] "GET / HTTP/1.0" 200 6397 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
I know the 200 code means it was a success
but nothing more, isnt it supposed to crawl ALL my pages? looks like it didnt crawl any.
So then I checked my logs for my subdomain (store.mysite.com) and found...
66.249.64.16 - - [28/Mar/2005:21:38:50 -0800] "GET /robots.txt HTTP/1.0" 404 - "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
66.249.64.16 - - [28/Mar/2005:21:38:57 -0800] "GET / HTTP/1.0" 200 14980 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
404 means error, didnt find the robots.txt in my [store.mysite.com...] Am I supposed to have a robots in my subdomain AND my main domain?
Go to [store.mysite.com...] ... if not found then the answer would be yes... you need a separate file.
Maj,
OK, thanks for clarifying that. However, should I ask my hosting service to add and edit the robots.txt file for me since I do not have one on my webstore which is a subdomain of my tld? Or is there another way to add one or perhaps resource(s) you can refer me to that will explain how to do this?
Jio