Welcome to WebmasterWorld Guest from 54.159.19.75

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
robots.txt question
I asked whether does this, but the engine spider ip address has any to use
2 ericchui 11:18 am Nov 22, 2004
Protecting robots.txt file from bad guys
Is it enough to passwoed protect those dirs?
3 damlag 4:10 pm Nov 21, 2004
How to Exclude from Yahoo/MSN results!
I want to exclude underconstruction site
2 fm101 1:49 am Nov 21, 2004
How to block all subdirectories?
2 TerryMc 7:05 pm Nov 17, 2004
How to exclude dynamic URLS?
So Search engines dont get bogged down.
7 Tech2004 4:54 am Nov 16, 2004
Effect of Robots.txt
5 seolancer 5:31 am Nov 15, 2004
robots.txt - newbie needs help
5 geezer_uk 5:39 am Nov 10, 2004
excluding cgi-bin
is this right
2 rbacal 2:59 am Nov 10, 2004
Is robot.txt a must for good SEO?
--whats the purpose of robot.txt
4 redleaf 12:24 am Nov 9, 2004
Weird 404 error hitting on robot.txt
anyone experienced this before?
3 redleaf 9:30 am Nov 8, 2004
robots.txt and free web hosting
3 rhodopsin 7:46 am Nov 8, 2004
Possible to have a page disalowed by robots.txt but indexed by adsense
Possible to do this?
2 brakkar 9:16 pm Nov 5, 2004
Protocol interpretation
Peerbot or I misunderstood the robot exclusion protocol
3 WebJoe 9:11 pm Nov 5, 2004
Need help about File Robots.txt
how do I know spiders visit to my site or not?
3 viva 10:15 am Nov 4, 2004
what happen with robot exclusion and iframes?
could it be a problem to exclude content that appears inside a home iframe?
2 mgorricho 5:03 pm Nov 3, 2004
protocol extension resource control
proposed extension to robots.txt protocol allowing resource control
1 copsewood 11:29 am Nov 2, 2004
Querystring check
2 papamaku 9:36 am Nov 2, 2004
robots and nav bars
excluding links in nav bars ...
2 f00fighter 11:38 pm Oct 29, 2004
Need assistance with robot controls
need help with a robot .txt file
2 borkboing 11:31 pm Oct 29, 2004
Robots.txt not working
bots not obeying robots.txt
4 cyberjunkie 11:05 pm Oct 29, 2004
Use .htaccess to serve correct robots.txt?
Using htaccess and robots.txt to elimiate file duplication and 404 errors
2 iceman 12:05 am Oct 26, 2004
Robots detection tool?
2 obsos 1:17 am Oct 25, 2004
robots.txt vs index,follow
Who will win?
3 brakkar 6:13 pm Oct 24, 2004
Removed 'deny all' robots.txt but no change
2 kapow 11:03 pm Oct 23, 2004
What name does msn's spider use?
Lots of visits from yahoo and google....but not msn
2 David_Bruning 7:05 am Oct 19, 2004