Spider Issues: Relative Links Confusing Spiders

A suggestion to use absolute links...

[JAB Creations]

I have noticed a large influx of really dumb requests by spiders. After some thinking (of why gamer pages where being requested from philosophy directory for example) I concluded that using relative links (../../gamer/file.php) was confusing some of the spiders.

I dont see issues with relative links that do not go outside their directory (href="file.php") but if you're like me and are trying to setup 301s and find all those nasty 404s and are having issues with dumb requests that you have no clue are being generated you can stop those by using absolute links.

Honestly I don't think we should have to be the ones to change but this problem is being generated by MANY spiders on my site as far as I can tell from my access logs, many of them without UAs (although one does check robots.txt) oddly enough!

Has anyone had this issue with any of their sites?

[wilderness]

Jab,
I've seen it happen is "spurts".
I haven't looked back in my logs for this reply, however off memory, I seem to recall it ne related to visitors using Linux. Not just SE's.

Don

[Dave_A]

From a search engine point of view, your relative links wouldn't be a problem for the web spider that we use.
Presently we run two, one will only stay inside a domain ( or Host ) whilst indexing and the other will multi thread and follow links outside the host.
Some of the robots coming into your domain don't sound Kosha and may be spambots.
I have heard of a web spider called Larbin or it may be Larpin that is being developed under GPL license, so it could be that someone is playing with a spider to see what it can do.

All the best
Dave A

[wilderness]

Larbin or it may be Larpin

Larbin.

It's in most every deny list ever created :)