Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Spiders get / and /robots.txt but no more!


dwilson - 8:55 pm on Dec 11, 2003 (gmt 0)


A service I was trying to use for my site-level search (not G, as I can't get them to re-index whenever I want) explained my problem.

"I took a look at your account and I noticed that your page has links such as
[MyDomain.com...] Because there is a slash missing from
the end to make the URL [MyDomain.com...] the spider
receiveds a re-direct to [MyDomain.com...] but cannot
follow the re-direct. "

That was the case for one spider, at least ... possibly more.

Hope this helps somebody else -- and thanks to those who gave me some ideas earlier.


Thread source:: http://www.webmasterworld.com/robots_txt/217.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com