Welcome to WebmasterWorld Guest from 54.242.63.214

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
does a search engine care if a robots.txt file is 403 instead of 404
4 kahuna 6:20 pm Mar 27, 2004
robots.txt and multiple domain names
Using a Virtual Server...
1 nerolabs 7:45 pm Mar 26, 2004
google images ignoring disallow
2 apollo 10:43 pm Mar 25, 2004
Validation Error
Ran the robots.txt validation checker and recieved nothing but errors.
3 jseamless 5:06 pm Mar 25, 2004
Why use Robots - Again?
What are the benefits of using Robots.
4 oltc 4:48 pm Mar 25, 2004
How can I block Cyveillancebot(63.148.99.****)
It doesn't look like a useful bot...
6 itisgene 4:43 pm Mar 25, 2004
Yahoo-MMCrawler
1 simondev3 8:08 pm Mar 24, 2004
Referral log spamming: How use robots.txt to keep them out?
10 Dan_Vendel 12:36 pm Mar 24, 2004
robots.txt elementary question
3 Vespasian 5:29 am Mar 24, 2004
How to identify search engine robots?
What's the best way for ASP developers to identify robots?
1 vieth 3:48 pm Mar 23, 2004
Robot text for dissallow asp pages indexed
2 DivinoStore 8:06 am Mar 21, 2004
Anyone recognize this agent?
Can't find any info about it anywhere
2 martyt 4:15 pm Mar 20, 2004
A new bot?
Anyone else seeing this?
4 walrus 6:48 am Mar 19, 2004
Can't prevent php files from being searched
Is this possible
1 Jack_mcs 1:58 pm Mar 18, 2004
newbie robots.txt and forum Q
how to write it
6 buksida 3:27 am Mar 18, 2004
Robot.txt File
What is it's importance?
7 Enigmatic 8:52 pm Mar 16, 2004
QuepasaCreep Agent
where is it from?
3 barbeta 8:17 pm Mar 16, 2004
Does robots exclusion prevent pr being passed
2 djtaverner 5:38 pm Mar 16, 2004
Should I ban this user agent?
4 jehoshua 5:13 pm Mar 16, 2004
syntax for all php exclusion
2 djtaverner 4:29 pm Mar 15, 2004
Cannot find error
Robots.txt syntax error
5 webapache 5:04 pm Mar 14, 2004
robots.txt and mod_rewrite
just double checking
6 hannamyluv 2:24 am Mar 14, 2004
Several robots.txt?
2 brakkar 1:18 am Mar 14, 2004
Where is the "Top Level"?
is this important these days?
4 chewy 9:46 pm Mar 13, 2004
Does this work?
Disallow: /somepath/*/file/*
6 DoppyNL 5:36 pm Mar 13, 2004

Featured Threads

Hot Threads This Week

Hot Threads This Month