Welcome to WebmasterWorld Guest from 54.166.5.230

Forum Moderators: goodroi

Message Too Old, No Replies

404 robots.txt and traffic going down

     
3:55 pm on Sep 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 20, 2003
posts:1741
votes: 0


Recently Iīve noticed 404 errors:

- -> /robots.txt

Seems like spiders are looking for that file. I never put robots.txt on my sites and thereīs no such a file in my /public_html/ directory.

buy I see traffic is going down and donīt know if it is because of that.

So what should I do?

3:57 pm on Sept 13, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 20, 2003
posts:408
votes: 0


Why dont you just add a robots.txt that allow all user agents access to everything - if thats what you want.
3:58 pm on Sept 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 3, 2003
posts:1633
votes: 0


If you don't need to block anything there is no need to have a robots.txt - it shouldn't be anything to do with your reduction in traffic.
4:00 pm on Sept 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 27, 2002
posts:1685
votes: 0


Check this page out as it explains most everything regarding robots.txt:

Robots.txt Validator [searchengineworld.com]

Pendanticist.

4:45 pm on Sept 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 14, 2002
posts:1192
votes: 0


The lack of a robots.txt file has nothing to do with the drop in your traffic.

Putting an empty robots.txt file will clean up your logs, it will have no other effect.

8:53 pm on Sept 13, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 3, 2003
posts:161
votes: 0


Put in a robots.txt file that contains:

User-agent: *
Disallow:

This will allow robots to view all files.

But usually there is at least one thing that can be disallowed.

4:20 pm on Sept 14, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 20, 2003
posts:1741
votes: 0


I did:

User-agent: *
Disallow:

But my site keeps tragically going down.
Past saturday (lowest day in visits) was 70
This is 12!

Iīm looking at google rankings and everyhting is ok...
What kind of problem would be causing this?
...

4:29 pm on Sept 14, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 14, 2002
posts:1192
votes: 0


Quite a few people seem to be experiencing similar problems, see [webmasterworld.com...]

Of course, others are not and the whole issue is very murky.

Let me return to the robots.txt issue. Google either indexes a page or does not, so a bad robots.txt file (or robots meta tag) could make your page(s) drop completely from Google. But if they are indexed, they are indexed :) The lack of a robots.tx file will not cause Google to knock a couple of points off your PR.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members