Brett's Spider Simulation

How are links really handled?

[thepcstore]

Hi.

When using Brett's Spider Simulator [searchengineworld.com], I realised that some of the links didn't work. This was because pages relative to the current document - such as 'filname.htm' - were seen by the simulation as 'http://filename.htm'.

Is this how the spiders really see things, or could it be a fault in this program?

It seems that Internet Explorer prefixes the links with the current site, so the above would then become 'http://www.example.com/filname.htm'.

Feel free to run the check on <snip domain> to see what I mean.

Cheers, Steve. :)

[edited by: engine at 11:23 am (utc) on Sep. 25, 2002]

[edited by: thepcstore at 11:48 am (utc) on Sep. 25, 2002]

[sw8296]

Just checked it for my own site and it works if you give the full file name:

[example.com...]

rather than

[example.com...]

[edited by: engine at 11:25 am (utc) on Sep. 25, 2002]
[edit reason] no specific urls, please [/edit]

[thepcstore]

Aaaaaah.....?

*scratches head*

But what does the spider see when it visits 'http://www.example.com'?!

By default the server should return 'index.htm' - which should be the same as entering 'http://www.example.com/index.htm'...

*thinks a bit more*

Steve. :)

[deltakits]

Actually, if you just type in http://www.example.com/ WITH the slash, it works fine. :)

[thepcstore]

Ooooh yeah!

Anyone know why it sees it differently? ;)

Steve.

[Brett_Tabke]

It doesn't follow redirects. Not all spiders will follow redirects.

I agree I should put a note in there for that situation.