Welcome to WebmasterWorld Guest from 54.144.107.83

Forum Moderators: goodroi

Message Too Old, No Replies

changing robots.txt

replacing robots.txt

     
8:02 pm on Dec 9, 2003 (gmt 0)

New User

10+ Year Member

joined:Sept 18, 2002
posts:23
votes: 0


If you put up a robots.txt with a disallow in it and then you want to reallow spiders - Is it sufficient to simply delete the robots.txt file or do you need to put up a new robots.txt file which explicitly allows all spiders everywhere (E.g.

User-agent: *
Disallow:

Ie is there some latency in the way spiders remember the robots.txt even after it has been deleted - if you don't specifically countermand the robots.txt with a new version?

(in particular I am interested in google)

8:24 pm on Dec 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 27, 2002
posts:1187
votes: 0


Safe thing is to replace it.

jb

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members