What if your domains robots.txt

Was pointed to your domain?

[Daily Sparring]

I have a site that was banned from Y and MSN. The sites Robots.txt (www.mydomain.com/robots.txt) would take you to the home page. So if you entered the above url, you would see the home page. Could this have been why I was banned? As Spam?

Please help...

[Terabytes]

the robots.txt....does NOT have the ability to redirect..

The redirect was most probabably done with a META refresh tag in the HTML...

(or with a MOD rewrite)

questions? 8-)

[Daily Sparring]

if it was done with a mod rewrite, would that affect the bots ability to spider or crawl.

[Terabytes]

the mod rewrite (in easy to understand terms...) is basicly a way of redirecting a user or robot from behind the scenes...(one of its functions, there are many...)

The user/robot interprets the page results as the genuine page it requested..even tho it's been redirected to another page.

(IE..the user/robot is looking for "pagename.htm", however that page no longer actually exists on the website, the mod rewrite can redirect the user/robot to another page without actually telling the user/robot that it was redirected, so it's not really looking at the page it requested but it doesn't really know the difference. The user/robot receives the same OK codes it would have received if it went to the originaly named page...)

(confusing?)

[jdMorgan]

An HTML page would be seen as an invalid robots.txt file format, so the 'bot could decide that either
A) It should crawl the site
B) It should not crawl the site
The interpretation could vary between different 'bots, and even between different revisions of the same 'bot.

Don't waste time worrying about it, just remove the redirect and continue looking for any other problems, such as multiple subdomains resolving to the same pages, i.e. more that just domain.com and www.domain.com resolving to the same content.

After any change to robots.txt, and even just periodically, it's a good idea to validate [searchengineworld.com] it.

Jim