Forum Moderators: goodroi

Message Too Old, No Replies

is there a reason to use a robots.txt

         

thorthedestroyer

4:47 pm on Oct 5, 2005 (gmt 0)

10+ Year Member



If I create a robots.txt will it do a better job at getting all pages indexed then simply having the following listed on every page:

<meta NAME="robots" CONTENT="index,follow,all">
<META NAME="GOOGLEBOT" CONTENT="INDEX, FOLLOW">

Thanks so much in advance.

robotsdobetter

2:22 am on Oct 6, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The robots.txt is used to block robots. The robots.txt and the Meta tags won't help you more pages spidered. If you want to get more web pages spidered get more backlinks.

duckxtales

8:41 pm on Oct 6, 2005 (gmt 0)

10+ Year Member



what are backlinks again?

Dijkgraaf

10:10 pm on Oct 6, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Backlinks are links pointing to your site from other sites.

stapel

2:50 am on Oct 7, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



To answer the question in your subject line, however, there is a reason to use a robots.txt file: Responsible robots check for it, so not having one will generate unnecessary 404 errors.

So even if your robots.txt file is empty (that is, you have nothing sensitive that you don't want good bots to stumble in to, so you've nothing to protect), having the file on the server will reduce your 404 error log.

Just a thought....

Eliz.