SEO Questions

[Enigmatic]

I have several questions concerning search engine optimization, which I'm hoping someone will be kind enough to answer for me.

First, let's say I built my main menu with SSI, so that changing a link name within the main menu could be done in one fail swoop. After all, no one wants to change content a 100 times over if it later needs to be updated, right? So can the SE spiders crawl an SSI menu? Is it safe or smart to do this?

Secondly, what is the general opinion on dynamic pages. Can SE's spider databases or scripts built with ASP, PHP or CGI? If I build a reciprocal links page with PHP or make a forum like this one, how easy is it for an SE to spider my site? If it's a bad idea, is there a way around this?

Lastly, where can I learn how to setup my robot.txt file and how important is this for SE spiders?

I would greatly appreciate anyone's help. Thanks in advance for your time,

Enigmatic

[Yidaki]

>So can the SE spiders crawl an SSI menu? Is it safe or smart to do this?

Yes, it's smart. Yes, it's safe.

SSI is processed on the server and the site get's delivered as ready made html code. The search engines' robots won't notice that you build your site using SSI's.

>Can SE's spider databases or scripts built with ASP, PHP or CGI?

Yes, they can. But to be on the safe side you should avoid using more than 1-2 parameters (par1=this&par2=that) or - better - mod_rewrite your url's to look like static url's (this-that.htm)

<added>and for your robots.txt questions - what jatar_k said while i was spellchecking my yadda yadda. :)</added>

[edited by: Yidaki at 9:12 pm (utc) on Feb. 23, 2004]

[jatar_k]

SSI

The ability, or inability, of a spider to crawl a menu has no bearing on SSI since the include happens on the server side and the spider is completely unaware of it's existence. All the spider receives is the output page after the includes are executed.

Dynamic Pages

The two major issues with dynamic site spidering are pages that take too long to load and long nasty query strings in the url. You will notice the urls on every page of this forum are nice, clean and easy to follow/spider.

and Robots.txt Tutorial [searchengineworld.com] and robots.txt Forum [webmasterworld.com]

[topr8]

let's say I built my main menu with SSI ... So can the SE spiders crawl an SSI menu? Is it safe or smart to do this?

anything processed server side is processed before the spider sees it, so the spider will not and can not know that it was created with a ssi or otherwise.

Can SE's spider databases or scripts built with ASP, PHP or CGI?

absolutely no problem but you may want to read on this forum more about this, generally keep the number of querystring attributes to a minimum and avoid using id=

Lastly, where can I learn how to setup my robot.txt file and how important is this for SE spiders?

not important unless you want to block spiders, robots text is about restricting access to your site not encouraging it, where to learn: maybe grab the WebmasterWorld one and have a look at it.

edit: gee everyone types so quick here, that was 3 similiar replies in no time!

[Enigmatic]

You're answers were all excellent and exactly what I needed. Thanks for the help! I have one more question though.

Thanks,

Enigmatic

[edited by: agerhart at 9:26 pm (utc) on Feb. 23, 2004]