homepage Welcome to WebmasterWorld Guest from 54.211.47.170
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
404 robots.txt and traffic going down
silverbytes




msg:1525622
 3:55 pm on Sep 13, 2003 (gmt 0)

Recently Iīve noticed 404 errors:

- -> /robots.txt

Seems like spiders are looking for that file. I never put robots.txt on my sites and thereīs no such a file in my /public_html/ directory.

buy I see traffic is going down and donīt know if it is because of that.

So what should I do?

 

PhraSEOlogy




msg:1525623
 3:57 pm on Sep 13, 2003 (gmt 0)

Why dont you just add a robots.txt that allow all user agents access to everything - if thats what you want.

dmorison




msg:1525624
 3:58 pm on Sep 13, 2003 (gmt 0)

If you don't need to block anything there is no need to have a robots.txt - it shouldn't be anything to do with your reduction in traffic.

pendanticist




msg:1525625
 4:00 pm on Sep 13, 2003 (gmt 0)

Check this page out as it explains most everything regarding robots.txt:

Robots.txt Validator [searchengineworld.com]

Pendanticist.

Mohamed_E




msg:1525626
 4:45 pm on Sep 13, 2003 (gmt 0)

The lack of a robots.txt file has nothing to do with the drop in your traffic.

Putting an empty robots.txt file will clean up your logs, it will have no other effect.

Ryan8720




msg:1525627
 8:53 pm on Sep 13, 2003 (gmt 0)

Put in a robots.txt file that contains:

User-agent: *
Disallow:

This will allow robots to view all files.

But usually there is at least one thing that can be disallowed.

silverbytes




msg:1525628
 4:20 pm on Sep 14, 2003 (gmt 0)

I did:

User-agent: *
Disallow:

But my site keeps tragically going down.
Past saturday (lowest day in visits) was 70
This is 12!

Iīm looking at google rankings and everyhting is ok...
What kind of problem would be causing this?
...

Mohamed_E




msg:1525629
 4:29 pm on Sep 14, 2003 (gmt 0)

Quite a few people seem to be experiencing similar problems, see [webmasterworld.com...]

Of course, others are not and the whole issue is very murky.

Let me return to the robots.txt issue. Google either indexes a page or does not, so a bad robots.txt file (or robots meta tag) could make your page(s) drop completely from Google. But if they are indexed, they are indexed :) The lack of a robots.tx file will not cause Google to knock a couple of points off your PR.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved