Welcome to WebmasterWorld Guest from 54.167.46.29

Forum Moderators: goodroi

Message Too Old, No Replies

Proper URL & File Structure

Fighting with Robots. Txt

     
9:00 am on Sep 19, 2007 (gmt 0)

New User

5+ Year Member

joined:Aug 2, 2007
posts:3
votes: 0


Hi Folks,

Thanks in advance, to all you guys who got some time to have a glance and end in helping me.

My question is while working with the URL Structure; Should the location of /.../ be anywhere on the URL for the Robots.txt OR Should the /.../ be exactly right next to the .com/..../

In a simpler way..My site creates Dynamic Pages with lots of redirects inbetween. So will this mess, messup with the Robots.txt file?

Please Help me ASAP.

Thans once Again.

1:55 pm on Sept 21, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


I don't understand the question.

However the robots.txt exclusion should specify only a filepath.

It should not specify the domain name within it.

The domain name is taken as baing the same as that used to access the robots.txt file itself.

12:34 pm on Sept 24, 2007 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3080
votes: 67


Hi srkmodi and welcome to WebmasterWorld!

You do not need to list the domain name in the robots.txt file. Other than the domain name you should list the entire file path (unless you are using wildcards aka pattern matching).

As for redirects you should avoid using multiple redirects. The search engines will follow most redirects unless blocked by robots.txt.

Hope this helps.