homepage Welcome to WebmasterWorld Guest from 54.226.191.80
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Blocking a file in the main folder?
Ept103




msg:1529148
 7:53 pm on Jan 22, 2005 (gmt 0)

The location to my file is [example.com...]

To block link.php for all robots, should I use this?

User-Agent: *
Disallow: /link.php

I'm not sure if I need the "/" or not after Disallow:?

 

LowLevel




msg:1529149
 12:35 am on Jan 23, 2005 (gmt 0)


I'm not sure if I need the "/" or not after Disallow:?

It is not strictly needed, but the most correct syntax includes the "/" char.


User-Agent: *
Disallow: /link.php

Correct. :)

Ept103




msg:1529150
 2:57 am on Jan 23, 2005 (gmt 0)

Ok ;-)

One more quick question:

link.php is what we use to count clicks..we have a function called url which when used in this syntax, counts the click then redirects to an outside site.

Example:

[example.com...]

Would the robots.txt in my message above prevent robots from serfing through any number of the above links? Or JUST the link.php file itself?

The reason I am doing this is because I don't want to be penalized for any of the links in my redirect should they show any errors, or be a "bad neighborhod" which could penalize me? Am I correct in the assumption that this does what I want it to?

mack




msg:1529151
 3:33 am on Jan 23, 2005 (gmt 0)

If you block link.php then that file can't be spidered, with ot without additional variables within the url (link.php?q=#*$!xxxx)

On order to spider any url it needs access to link.php if you have blocked access to that then they cant go any further.

Mack.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved