Forum Moderators: open
I was looking around at the other posts and realized I don't have a clue about spiders. It seems a lot of you are dissatisfied and don't want them.
From my standpoint...That is the only way I appear in the search engines because I am still working on my site and haven't submitted to any yet. I am getting some excellent rankings for some of my phrases.
My site only has text and clipart so I would not have to be concerned with images etc.
Could someone give a brief summary on things I need to know?
One problem that I was having was with my free hosting banner. All the search engines have Click Here Free Web Hosting By (my host name) This site is hosted by... instead of what I put as my site title. After the first time I uploaded, AltaVista had "my" title and description but now it has some wording that seems to have come off my coming soon starter page before I got going.
A web designer told me the host banner code is positioned immediately after the <html> tag in my page code and that many search engines grab the first text they find and plug that in as the title/description. They said for my title to show in the search results I would need to move the banner code further down the page.
I decided to remove the banner all together. Will the search engines pick up on my title and description now that the banner is gone (if so, I wonder when)?
Thanks for your help!
The problem is not that spidering is bad. The problem is that too darn many crawlers/spiders are extremely badly behaved. Consequently, many of us have ended up configuring our sites to allow some spiders in but too keep others (probably more) out.
My own site gets very significant traffic from Google, and I'm happy to report that Google's staff is very responsive if you e-mail them about problems with their crawler.
Other services, though, couldn't give a flip whether or not they cause your site problems when they crawl it, and good luck getting a response from the staff, assuming you can even figure out what service the crawler is associated with and how to contact the people running it.
This latter point is important because many crawlers are not associated with any serach engine I've ever heard, and many/most of them may be crawling websites for e-mail addresses they can feed into spam databases.
"A web designer told me the host banner code is positioned immediately after the <html> tag in my page code and that many search engines grab the first text they find and plug that in as the title/description. They said for my title to show in the search results I would need to move the banner code further down the page. "
By title, I assume you mean the title in the <body> of the page and not in the <head>. In which case, that's generally correct.
"I decided to remove the banner all together. Will the search engines pick up on my title and description now that the banner is gone (if so, I wonder when)?"
They'll pick up on it, but it's a carpshoot as to when. Some crawlers might figure it out tomorrow, some next month. It depends on what sort of "refresh" cycle they've been programmed with. If your site is new, they may have you marked down for infrequent refresh. You can try using the submit utility on some of the major bots to give them the hint that they should come around sooner than later, but there's no guarantee that it will be successful.
