homepage Welcome to WebmasterWorld Guest from 54.146.175.204
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Blocking Partial Contents of Folder Without Using Robots Tag
10 files (7 want indexed, 3 want blocked)
SandySEO

5+ Year Member



 
Msg#: 3290795 posted 9:47 am on Mar 23, 2007 (gmt 0)


System: The following message was cut out of thread at: http://www.webmasterworld.com/robots_txt/3274413.htm [webmasterworld.com] by goodroi - 6:24 am on Mar. 23, 2007 (utc -5)


Hello Friends,

Can you please help me out. I have 10 files contained in a folder. I want to crawl 7 files and dont want index remaining 3 files.

User-agent: *
Disallow: /folder/
- I guess it will not index all the files contained in this folder.

I only want to do this through robots.txt files not with robots meta tags.

I am looking for your replies friends.

Thanks in advance,
- SandySEO

 

SandySEO

5+ Year Member



 
Msg#: 3290795 posted 10:16 am on Mar 27, 2007 (gmt 0)

Should I guess that no one is capable to give me the answer of my query?

WW_Watcher

5+ Year Member



 
Msg#: 3290795 posted 2:05 pm on Mar 27, 2007 (gmt 0)

SandySEO, with a name like that, why are you asking us? Simple, basic, beginner stuff for an SEO expert.

Block each file by name that you wanted blocked in the robots.txt.

Back to Lurking
WW_Watcher

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3290795 posted 7:53 pm on Mar 27, 2007 (gmt 0)

I only want to do this through robots.txt files not with robots meta tags.

Robots.txt was built to block entire directories and not individual files. I would suggest you rethink about using robots meta tags.

Key_Master

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3290795 posted 8:49 pm on Mar 27, 2007 (gmt 0)

User-agent: *
Disallow: /folder/file1.ext
Disallow: /folder/file2.ext
Disallow: /folder/file3.ext

[robotstxt.org...]

[edited by: Key_Master at 8:51 pm (utc) on Mar. 27, 2007]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved