Welcome to WebmasterWorld Guest from 54.234.114.182

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
robots.txt file being ignored and old content not being removed
10 Marty3454 11:46 pm May 20, 2008
To use or not to use a robots.txt file
3 con771 6:03 pm May 20, 2008
4 Wordpress File's Link To Be Blocked
How should I block them?
5 aced84 3:54 pm May 19, 2008
I think there is a problem
Robots problem
2 AllanS 10:39 am May 17, 2008
robots.txt syntax for wildcards?
3 Marfola 7:15 am May 15, 2008
How many links can i place in one page of site map?
5 bava_seo 2:12 pm May 10, 2008
why use and what is robots.txt
this maybe stupid question
3 mick_simon 4:45 pm May 7, 2008
How To stop crawler to crawl https:// pages
8 vreelmistee1 6:03 am May 7, 2008
Status: 404(not found) in Google Webmaster Tools
3 Boulder90 6:39 pm May 6, 2008
Y! and robots.txt
Partial ignore
5 Staffa 6:36 pm May 6, 2008
Googlebot is Not Taking Visit to my site
robots.txt is restricted googlebot to do not visit
4 bilalseo 1:54 pm May 6, 2008
Site Pages are Blocked by Robots.txt
4 bilalseo 10:03 am May 6, 2008
robots.txt and IP addresses
11 Maciej_Ziemczonek 10:11 pm May 2, 2008
Is this valid? wildcard in robots
Disallow: /*/edit.html
3 rudyten 1:47 pm Apr 20, 2008
Will spiders follow links on URL's blocked by robots.txt
3 deanw 6:34 pm Apr 16, 2008
Security Challenges
4 pageoneresults 3:49 pm Apr 2, 2008
Dumbest question eva: Do all major SEs support the Google guidelines?
I am testing mine on G's WebmasterCentral
2 walkman 6:42 pm Mar 26, 2008
Is there any way to block everyone except.
for Google, Yahoo etc?
2 mmiller 9:41 am Mar 21, 2008
need to block. and don't know - pls help
need help with blocking - Malware Ref
4 Reilly 7:55 pm Mar 17, 2008
How to block original + localized pages?
robots.txt localized
1 vanguard2000 6:19 pm Mar 17, 2008
How long before bots re-visit after they were told they are not allowed?
3 boromir27 3:32 am Mar 17, 2008
Inktomi User-Agent blocked
Inktomi blocked
6 cyberdyne 8:36 am Mar 14, 2008
robots.txt - Disallow/Allow
7 eyalkattan 1:14 am Mar 13, 2008
robots and redirecting
i am wondering how a robot would act on a page with a redirect
2 melanger 8:56 pm Mar 8, 2008
Unusual log entries
5 cyberdyne 6:17 pm Mar 8, 2008

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members