Welcome to WebmasterWorld Guest from 54.226.241.8

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
where to place robots.txt?
robots newbie alert
6 Duke_of_Url 10:17 pm Jun 10, 2004
How should I use a robots.txt file on a virtual account?
I want to ban spiders from one site only
3 soccer_star 2:01 pm Jun 10, 2004
Robots.txt for yahoo and google.com
google and yahoo robots.txt
3 adisakmai 4:47 am Jun 10, 2004
Prevent Google's "cached" link from appearing
but still stay in the search results
4 digitalv 4:51 pm Jun 8, 2004
Disallowing specific query strings
3 duke33 3:05 pm Jun 8, 2004
Is it a spider?
2 topher 5:38 am Jun 8, 2004
Preventing robot access to all subdirectories within a directory
2 Leonardi 12:16 am Jun 7, 2004
Robots.txt triggers 404 error
Robots.txt triggers 404 error
2 Nout 11:10 am May 29, 2004
Is this robots.txt ok?
5 esllou 12:32 pm May 26, 2004
Removing "query-stringed" pages from Google
And other search engines using robotx.txt
3 MatthewHSE 2:23 pm May 25, 2004
Stop Spiders Crawling Message Board Edit
Cgi Factory Message Board
7 mn1dbp 12:36 pm May 25, 2004
Google still lists disallowed pages
How long do disallowed pages stay in the index?
5 jam13 10:56 am May 23, 2004
large robots.txt file
What size will cause a crawler to choke in the file
7 DoppyNL 9:43 pm May 21, 2004
Using robot.txt to keep unwanted backlinking
Is this an effective means
2 Daily_Sparring 5:14 pm May 21, 2004
importance of a robots.txt file
11 hughie 12:27 am May 21, 2004
Logic Error
Wildcard userAgent last entry in robots.txt file?
2 pageoneresults 2:34 pm May 20, 2004
Question about robots meta tag, as well as robots.txt....
3 xsirrionx 11:48 am May 20, 2004
How to validate robots.txt on "https" site?
Please help me validate my "https" site
2 CodeCrunch 12:29 am May 20, 2004
Wild chars *.* allowed in "Disallow" of Robots.txt?
or anything else like *.txt, etc.
2 CodeCrunch 12:26 am May 20, 2004
robots.txt restrictions conflictual with site's Terms
Which of them should prevail?
4 cristiscu 6:52 am May 19, 2004
Comments at the end of a line bad style, why?
3 zachariah 6:52 am May 18, 2004
Spider Testing
6 cabowabo 3:44 am May 13, 2004
Meta Tags & Robots.txt
4 tcustom 1:57 am May 13, 2004
Why is Google still indexing my SWF files?
2 ichthyous 6:10 am May 10, 2004
Deny or Not to Deny /images
Denying /images folder
9 quebrada 12:19 am May 6, 2004

Featured Threads

Hot Threads This Week

Hot Threads This Month