Welcome to WebmasterWorld Guest from 54.145.209.34

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Google CHTML Proxy disobeying robots.txt
Why?
5 yosmc 12:16 pm Aug 31, 2004
Google tells me to change my robots.txt
Should I follow the advice from Google adSense?
2 Bonusbana 2:44 pm Aug 27, 2004
Master List of Malicious Robots
Is there an active list of the bots we should block?
5 Eric_Lander 4:09 pm Aug 25, 2004
robots.txt security problems?
2 helohelo 3:44 pm Aug 25, 2004
301 Redirect domain?
How to handle robots.txt?
5 Mr_Roberto 1:13 am Aug 24, 2004
Creating robots.txt file.
6 kabanero 8:34 pm Aug 18, 2004
Blocking just one parameter via robots.txt
Is this even possible?
2 werty 7:31 pm Aug 18, 2004
Problem with spiderbots scrounging for robot.txt
3 Mr_N 6:49 am Aug 10, 2004
i have a subdomain I dont want indexed
is robots.txt the answer?
4 chrisandsarah 1:21 pm Aug 8, 2004
robots.txt appearing in my log files
But I have n't got one there!
5 uk_webber 2:59 pm Aug 5, 2004
To Robots.txt or not
Do we use both?
4 obsos 11:20 pm Aug 1, 2004
msn search violating robots.txt!
`disallowed` images appear in preview on result page
26 DoppyNL 4:15 pm Aug 1, 2004
Robots text file or noindex,nofollow
Whats better?
4 mthorpe 4:18 am Jul 31, 2004
google allow - disallow all others
6 jimmy19 7:20 am Jul 30, 2004
MSN and Yahoo both missing robots.txt
4 bppilot 3:26 pm Jul 29, 2004
Yahoo robots.txt problem
what redirect?
4 yowza 9:53 pm Jul 27, 2004
Google listing disallowed pages
this normal?
2 sonik 6:44 pm Jul 22, 2004
robots.txt for different language sites
can use charset tag to allow/disallow?
4 a2ztranslate 11:51 pm Jul 20, 2004
about subdomain and robots
should i copy robots to all subdomains?
2 webon 3:33 am Jul 18, 2004
robots.txt question
3 motorcycle 5:30 pm Jul 17, 2004
Disallowing Robot's Effect on Sub-Domain
Will it disallow the sub domains as well
2 bppilot 5:53 am Jul 14, 2004
Why is Google following my exclusions?
Robots.txt file says dont go and Google does!
5 anxvariety 5:49 am Jul 14, 2004
Why are my .swf files still being indexed by Googlebot?
3 ichthyous 6:26 pm Jul 13, 2004
excluding dynamic pages in all search engines
excluding dynamic pages in robots.txt
3 toddmc04 2:49 pm Jul 13, 2004
General Guide to Robots.txt
is there one here :)
4 rj87uk 7:51 pm Jul 7, 2004

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month