Forum Moderators: open

Message Too Old, No Replies

what makes a site spiderable?

         

hokiss

10:29 pm on Jul 7, 2003 (gmt 0)

10+ Year Member



i want to ask what makes a site spiderable?

jeremy goodrich

10:30 pm on Jul 7, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If a search engine can reach the domain in question & parse the contents.

Symbios

10:33 pm on Jul 7, 2003 (gmt 0)

10+ Year Member



Links from other sites will let the spiders in.

A good linking structure on your site will let the arachnids explore further.

hokiss

1:03 am on Jul 8, 2003 (gmt 0)

10+ Year Member



well i have a site writen in asp mssql it use to get all site info but now the spiders arent getting the web sites that are listed
why?

agerhart

12:45 pm on Jul 8, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



hokiss,

Does your website use session IDs?
Do your URLs have numerous special characters within them, such as &,?, etc?

hokiss

2:53 pm on Jul 8, 2003 (gmt 0)

10+ Year Member



yes it has and ive send you message

shasan

4:14 am on Jul 9, 2003 (gmt 0)

10+ Year Member



Hi all,
I'm new to this board and the whole SEO thing. Wondering if someone can help me out. I have a related question:

a partner site wants to link to me using a javascript popup. Would the spider see the URL within javascript:popup('http://www.mysite.com') as a reference?

Thanks.
Sameer.

thewebboy

5:47 am on Jul 9, 2003 (gmt 0)

10+ Year Member



I have finished setting up my little search engine and I come across a few things that hold on my little bot. Here are a few of the problems:

1) blocking because of user-agent string
2) blocking for not having javascripts installed
3) Cookies required
4) Meta tag refreshes instead of 302's