Big problem with ASP pages and SQL Server

ASP, SQL Server, Microsoft OLE DB Provider for ODBC Drivers error 80040e21

[elguiri]

A client of mine has a dynamic directory site in ASP running off a SQL Server database. When I request pages in IE or in Mozilla everything works fine. When I test the pages with searchengineworld's Sim Spider (an excellent tool) I get either:

1. no results for the parameters in the URL (when I know such results exist) OR, more commonly,
2.
Microsoft OLE DB Provider for ODBC Drivers error '80040e21' ODBC driver does not support the requested properties. /filename.asp , line 213

As the pages are generated server-side, I have no idea as to why the error is produced only with certain clients (like a search engine bot).

Any help would be very much appreciated. (PS. I did tell them not to use ASP!)

[CaseyRyan]

Could the pages be relying on Session variables?
I don't know what kind of support spiders have for an ASP session. The session uses cookies which I'm not sure that a spider would support. I guess that might even be per spider.

Another idea is that their display is based on the useragent of the browser/spider?

Do you have any visibility into the code?

-=casey=-

[Easy_Coder]

I think Casey gets a bingo on the session suggestion. Do you have any http header tools that you can surf the site with and watch what's going on? I know MS has a free one called Fiddler; just g for fiddler (should be the 1st result).

[elguiri]

Thanks for your replies.

I have the code for some of the pages but am not sure which bits are relevant to post. What makes me think it's not a session problem is that it doesn't happen for all pages.

It happens when multiple parameters passed through the URL. The bot seems to get stuck with the ampersand.

Is this because of the simplicity of the bot? Will Google, Yahoo, or MSN have the same problem? Would an ISAPI rewrite sort this?

Thanks again.

[Easy_Coder]

elguiri it sounds like your bot has limited or no support for querystring parameters. You won't have that problem with G, Y or M.

[elguiri]

Thanks Easy_Coder. That's a weight off my mind.

Three further questions:

1. Given that the bot in questions is the searchengineworld sim spider, do you think Brett could be persuaded to upgrade?

2. Do you know of an available bot simulater that supports querystring parameters in a similar manner to the large search engines?

3. Nowadays, would you go down the ISAPI URL rewriting route anyway? Is cost (time) outweighed by the benefits?

Thanks again for all help.