Forum Moderators: open

Message Too Old, No Replies

robots.txt to be or not to be?

ive read so many conflicting comments can we clear it up?

         

humpingdan

4:17 pm on Oct 9, 2003 (gmt 0)

10+ Year Member



this is my robots.txt

User-agent: *
Disallow: /

i think this allows all robots to index my site apart from /images folder? yes?

so should i have this on my server or not, i want to allow all robots but some people say if i want to allow all i should just not have a file there!?

help me somebody!

ciml

5:27 pm on Oct 9, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I think you mean "Disallow: /images"; if so then yes that forbids only URLs beginning with /images

Google can list those URLs, but it won't fetch them so there will be no title, description or cache. Just a link with the URL as anchor text in the SERPs.

jatar_k

5:31 pm on Oct 9, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



User-agent: *
Disallow: /

that would mean

all user agents are not allowed to access anything on your site, as ciml said

User-agent: *
Disallow: /images/

that would mean all user agents are not allowed to access anything in the images folder which is located at the root level of your site.

you could take a look at this as well
Robots.txt Tutorial [searchengineworld.com]

dirkz

6:10 pm on Oct 9, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I would just remove the file. Then there's no problem. Having images in the index can help :)

bignet

11:55 pm on Oct 9, 2003 (gmt 0)

10+ Year Member



dirkz, how does it help
bots banned from my image pool for bandwidth saving
and rankings are very well