homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Robot.txt file deleted by accident what now?

10+ Year Member

Msg#: 515 posted 10:48 pm on Dec 12, 2004 (gmt 0)

Sorry everyone, to bore with a repetetive question.

I have been handed a site(with little experience myself) changed a few things including the page title on index page, (using frontpage 2003) added a few keywords to site content pages......then uploaded new site contents then dead..no listings anymore in google... whereas there used to be many! Gulp!

I have waited for over five weeks in the hope that my new revamped (or cocked up) site would be re-indexed but nothing. In desperation I have removed robots.txt file as it read as follows...

# for use by all robots
User-Agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /scripts/
Disallow: /styles/
Disallow: /Thumbs.db

And I think I may have totally misundertood everything here. I thought this meant that none of my site would get read?

Did I do wrong and should I put this back, please help going mad day by day.




WebmasterWorld Senior Member 10+ Year Member

Msg#: 515 posted 11:15 pm on Dec 12, 2004 (gmt 0)

If you have no robots.txt then robots will feel free to look at anything they want.

Your previous robots.txt excluded robots from much of your website. If all of your pages were in the excluded directories, they may have concluded you didn't like company and are politely keeping their distance.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved