homepage Welcome to WebmasterWorld Guest from 184.73.74.47
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Pubcon Website
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library : Charter : Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Proper URL & File Structure
Fighting with Robots. Txt
srkmodi




msg:3454670		 9:00 am on Sep 19, 2007 (gmt 0)

Hi Folks,

Thanks in advance, to all you guys who got some time to have a glance and end in helping me.

My question is while working with the URL Structure; Should the location of /.../ be anywhere on the URL for the Robots.txt OR Should the /.../ be exactly right next to the .com/..../

In a simpler way..My site creates Dynamic Pages with lots of redirects inbetween. So will this mess, messup with the Robots.txt file?

Please Help me ASAP.

Thans once Again.

 

g1smd




msg:3457139		 1:55 pm on Sep 21, 2007 (gmt 0)

I don't understand the question.

However the robots.txt exclusion should specify only a filepath.

It should not specify the domain name within it.

The domain name is taken as baing the same as that used to access the robots.txt file itself.

goodroi




msg:3459158		 12:34 pm on Sep 24, 2007 (gmt 0)

Hi srkmodi and welcome to WebmasterWorld!

You do not need to list the domain name in the robots.txt file. Other than the domain name you should list the entire file path (unless you are using wildcards aka pattern matching).

As for redirects you should avoid using multiple redirects. The search engines will follow most redirects unless blocked by robots.txt.

Hope this helps.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved