Welcome to WebmasterWorld Guest from 54.163.100.58

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Robots.txt command? "do not search anything outside this."
5 OWizeOne 1:08 pm Feb 1, 2007
How do I block yahoo's SE spider and allow their image spider?
Does Yahoo send a different spider for the image search?
4 Clark 5:18 pm Jan 27, 2007
How robot.txt works
Robot.txt
3 walmslei 4:10 am Jan 27, 2007
Why is google indexing this banned directory?
5 ichthyous 4:55 pm Jan 26, 2007
https on google and robots.txt
2 concepts99 1:54 pm Jan 24, 2007
directory structure question
5 union_jack 7:43 pm Jan 22, 2007
block a subdirectory
2 concepts99 2:05 pm Jan 22, 2007
Google not respecting robots.txt
Found >500 pages in Google index that are forbiden by robots.txt
5 roots 5:54 am Jan 22, 2007
domo arigato
mr. roboto
3 ricnut 11:52 pm Jan 20, 2007
Disallow Slurp from all files except two
Yahoo likes only one page, so I'd like to just 'give' it that.
2 tigertom 3:00 am Jan 20, 2007
Yahoo Slurp Bot misbehaving
.. or is it someone masquerading?
4 tigertom 9:27 am Jan 17, 2007
Blocking Site in some countries
3 eldinboy 5:55 pm Jan 15, 2007
What to do with duplicate pages
3 olba 2:12 am Jan 12, 2007
Why should I have a robots.txt file?
Isn't it OK if I don't?
23 beren 8:45 am Jan 11, 2007
Google Site Diagnostics - robots.txt blocks urls
4 akreider 7:44 pm Jan 5, 2007
Robots txt to solve gallery duplicates
can someone confirm this is correct
3 caines 10:54 am Jan 5, 2007
How many search engine understand crawl delay?
1 foxfox 3:38 pm Jan 4, 2007
Multiple Domains Issue
3 kdobson99 4:58 am Jan 4, 2007
Do spiders follow links from pages with the noindex tag?
As opposed to links from robots files?
4 bouncybunny 6:59 am Jan 2, 2007
Directing Spiders to specific Servers
2 TheSeoGuy 1:50 pm Dec 29, 2006
I had robots.txt blocking my phpbb folders which are now deleted
Should I remove that line from robots.txt now?
2 cmendla 4:16 pm Dec 24, 2006
Disallowing Indexed files in Google
5 leo11877 2:01 pm Dec 23, 2006
Disallowing by date.
Is it possible to keep some page and disallow new ones...?
3 bouncybunny 8:31 am Dec 20, 2006
Worried about profile links
4 Asia_Expat 3:53 pm Dec 19, 2006
Is it possible to exclude a URL by specifying an anchor?
Excluding #...?
3 bouncybunny 12:40 am Dec 17, 2006

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month