Do I need a robots.txt file for Googlebot to read?

Will I get indexed by Google if I don't have a robots.txt?

[johnd]

googlebot hit my page a couple days ago and requested the robots.txt file. I don't have it.

does it have to be there? what if it's not?

thanks

[SmallTime]

Hi and welcome to WWM. robots.txt does not have to be there, but it is a very good idea. See [webmasterworld.com...]

[johnd]

thanks for your reply. let me make sure I get this right. i want to make sure that googlebot (and other spiders) index my site.

is it better to put a robots.txt file on my server or not to put it?

[SmallTime]

Much better to put it there, but make sure that you have the syntax right.

[Mardi_Gras]

Johnd - there are a few threads on this subject here. In addition to the one cited by SmallTime, check out:

[webmasterworld.com...]
[webmasterworld.com...]

[Marcia]

It's worth repeating - even having a blank page named robots.txt will avoid a lot of 404s.

[Mardi_Gras]

That's the reason I finally put one up, Marcia - and, just as you suggested, it is blank. Maybe one day I'll get around to excluding someone!

[deejay]

The mention of blank robots.txt files is giving me the heebies...

I've read more than once never to use a blank robots.txt as some spiders will interpret it as a 'disallow all'.

Have I been led up the garden path?

[Mardi_Gras]

deejay - all a robots.txt file does is tell robots where they cannot go, and there is a standard (robot exclusion standard) for the language to be used. A blank page does not convey any "disallow" information; it just stops a 404 error from showing up in your logs.

[johnd]

there seems to be no one answer to the question of robots.txt

it wouldn't be logical if not having one would be interpreted as disallow all, because a large number of web sites would never get a chance to be spidered (people who never heard of robots.txt - personal home pages etc would never make it to the web). it doesn't mean it's not true on the other hand.

[nutsandbolts]

there seems to be no one answer to the question of robots.txt

There is one answer ;) Just put a blank robots.txt file - and everything will be crawled fine. And because we are in the Google forum, the most amazingly witty, funny and intelligent GoogleGuy told us to do it!

[edited by: nutsandbolts at 10:42 am (utc) on June 17, 2002]

[johnd]

GoogleGuy told us to do it!

then, I'll do it :)

[bill]

Here's the GoogleGuy robots.txt thread [webmasterworld.com] for those interested...

[deejay]

*scrawling on yet another post-it and slapping it onto a corner of the screen* ok.. ya got me.. I'll put one up.

Thanks for the responses :)

I am ever more impressed with this place.

[johnd]

I am definitely impressed with this forum :)

[wangdy]

Hey everyone...
along the same lines of this thread... where does the robots.txt need to reside?

Does that mean that one .txt is recommended for each URL/server, etc?

[SmallTime]

In the root directory of the website (same place as your initial index page)

[Mardi_Gras]

To check your robots.txt, use Brett's handy validator:

[searchengineworld.com...]

[josh]

just curious, should the (blank) robots.txt be uploaded to the root such as www.domainname.com/robots.txt or where the site is www.domainname.com/mysite/robots.txt or both?

may seem like a stupid question, but it's been itching. thanks!

[agerhart]

The robots.txt should be in the root directory. From this spot in the root, you will define what the robots can and can not access. If you want them to reach a folder within the other sub-section of your site, allow access, if not, deny the robot access.

[josh]

thanks agerhart... so a blank robots.txt will just allow all access?

[agerhart]

It shouldn't be blank, but you shouldn't disallow access if you want them to roam free.

[johnd]

ok, so now it shouldn't be blank.

what do I put in there then?

"allow all" or what?

[Mardi_Gras]

johnd - with all due respect to agerhart, I don't think it matters if it is blank :) But Brett has an article that will tell you everything you need to know about preparing a properly-functioning robots.txt at:

[searchengineworld.com...]

[agerhart]

I think I may have not clarified what I meant, and I think it comes down to the way that you like to set yours up.

I always list out all of the important spiders and robots in my robots.txt and then specify if they have full access, partial access, or no access at all.

In my opinion, this makes it easier to change it in the future.

[johnd]

ok, cool. I want all spiders to spider everything they want, and have a good time on my sites

I'll leave them a blank robots.txt :)

[Mardi_Gras]

Or just drop in:

User-agent: *
Disallow: