homepage Welcome to WebmasterWorld Guest from 50.17.66.61
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
how to block a page
on a particular location
experienced

10+ Year Member



 
Msg#: 3420667 posted 11:04 am on Aug 13, 2007 (gmt 0)

i want to block a particular file located under the various folder with the same file name.

domain.com/folder1/file.php
domain.com/folder2/file.php
domain.com/folder3/file.php
domain.com/folder4/file.php

I have approx 15000 folders like this where i want to block the file.php in every folder. what is the best option to block. I dont want to make the robot file so lengthy by putting all the urls in side. any other quick solution.

help appreciated in advance :-)

[fixed confusing typo in title]

[edited by: goodroi at 1:54 pm (utc) on Aug. 13, 2007]

 

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3420667 posted 1:02 pm on Aug 14, 2007 (gmt 0)

An easy way to handle this with the big search engines is to use wildcards aka pattern matching. IMPORTANT This is not supported by most other bots.

Google Robots.txt Pattern Matching Explained
[google.com...]

Yahoo Robots.txt Wildcards Explained
[ysearchblog.com...]

experienced

10+ Year Member



 
Msg#: 3420667 posted 2:20 pm on Aug 20, 2007 (gmt 0)

will this expression block this file at any level.

User-Agent: Googlebot
Disallow: /my-file-name.php*

domain.com/folder1/my-file-name.php
domain.com/folder2/my-file-name.php
domain.com/folder3/my-file-name.php

if yes then this would be really helpful for me.

Thanks

experienced

10+ Year Member



 
Msg#: 3420667 posted 4:49 am on Aug 22, 2007 (gmt 0)

pls reply if possible

new_seo

5+ Year Member



 
Msg#: 3420667 posted 5:39 am on Aug 22, 2007 (gmt 0)

If you want to block a particular file which is located under the various folder with the same file name,then I think you can do it simply by
User-Agent:*
Disallow: /file.php

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3420667 posted 6:01 pm on Aug 22, 2007 (gmt 0)

This will block Google from all filenames containing "my-file-name.php"

User-Agent: Googlebot
Disallow: /*my-file-name.php*

Please remember that most smaller search engines do not support this in robots.txt. Also if you want to test other combinations for Googlebot you can go to Google's Webmaster Central and use their robots.txt analysis tool.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3420667 posted 9:47 am on Aug 24, 2007 (gmt 0)

This will disallow any URL that begins /file.php....

Disallow: /file.php

... and that only works for URLs in the ROOT, i.e. BEGIN with that.

.

You need the * to make it work for folders.

Disallow: /*file.php

This Rule MUST go in the User-Agent: Googlebot section.

Other bots do not understand the *.

.

If you have a User-agent: Googlebot section, then ALL of your rules for Googlebot must go in that section as Googlebot will then completely IGNORE the User-agent: * section.

You do this even if it means duplicating a lot of stuff into both sections.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved