Welcome to WebmasterWorld Guest from 34.238.194.166

Forum Moderators: goodroi

Message Too Old, No Replies

possible sabotage?

robot.txt working against me?

     
7:15 pm on Sep 8, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Feb 21, 2002
posts:228
votes: 0


I was hired to provide SEO services to a catalog site in .aspx.

The designer gave me limited access to only a few directories.

I fully expected to be seeing certain things by this point into the campaign. I checked the Lycos InSite submission manager and it noted "disallowed" by robot.txt.

Here is a partial of the robots.txt -

# FULL access (Google)
User-agent: Googlebot
Disallow: /announcements/
Disallow: /cards/
Disallow: /paper/
Disallow: /print/

# FULL access (Inktomi)
User-agent: Slurp
Disallow: /announcements/
Disallow: /invitations/
Disallow: /paper/

# FULL access (Lycos)
User-agent: Lycos_Spider_(T-Rex)
Disallow: /announcements/
Disallow: /cards/
Disallow: /invitations/
Disallow: /print/

# FULL access (All Spiders)
User-agent: *
Disallow:

Now, my SEO folders are the ones listed next to disallow..........what would any of you think of this?

Too - what is the purpose of the last block?
# FULL access (All Spiders)
User-agent: *
Disallow:

8:09 pm on Sept 8, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
posts:1179
votes: 0


The first three are saying disallow specific spiders to certain directories and the last one is to disallow none and give all robots complete access

Why?

I have not seen Lycos_Spider_(T-Rex) in years, last time I checked Lycos were using fast alltheweb database which is now part of yahoo

ncw164x

8:22 pm on Sept 8, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Feb 21, 2002
posts:228
votes: 0


I don't know the designer personally - so I'm not sure why the robots.txt was placed the way it was or why it has what it has in it. I imagine his primary function is that of a designer and not much else. (Good design though).

I just saw that it appeared that my SEO directories were being blocked by it.

Thanks

9:27 pm on Sept 8, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 8, 2004
posts:196
votes: 0


The lines

Disallow: /announcements/
Disallow: /cards/
Disallow: /paper/
Disallow: /print/

will ban those robots from accessing those directories. If those are the directories holding that pages that you are SEOing, then you have a problem.

9:36 pm on Sept 8, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Feb 21, 2002
posts:228
votes: 0


Right - I just wanted to verify what I thought was the case. Since I don't have access to the files site-wide, and I had suspicions about the robots.txt file (which I can't see via FTP), I decided to try looking up ["domain".com...] - that's how I found out.

Thanks again. Soon - there should be no more problem ;-)