Forum Moderators: open
very new to all web design, just put first site live for my driver training business, and now working on much better one and getting to grips with external css.
anyway, just wanted to know how you can find out which spiders/bots (whatever) are searching you, and how they are indexing.
I only have had my site live for 4 weeks and have google rank 4 but would like to find out more info to add to learning curve.
thanks
geoff
ps: do any experts know where i can get hold of a script to run 4 images same size as in a slideshow but they rotate automaticaly and also fade in and out from each other? I created the ffect in flash but would like to create what i want away from flash.
cheers
anyway, just wanted to know how you can find out which spiders/bots (whatever) are searching you, and how they are indexing.
When your website is accessed, the requestor (whether it is a human or spider) sends some information to the server called the "User-Agent". The User-Agent identifies the requestor. If it is a human, the User-Agent will be that of a web browser, something like "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.5) Gecko/20060719 Firefox/1.5.0.5"
Spiders have User-Agents, too. For example, Google's spider has a User-Agent of: "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
If you see a user agent that looks like it might belong to a search engine, that's one way of identifying the spiders.
Another way is to compare the IP address of the visitor to a list of known search engine spiders.
Many webmasters set up "bot-traps", which are basically web pages that are linked to from the home page. The link's anchor will be something that a human would never click on. A spider would follow that link, though, and therefore be identified.
Sometimes the bot-trap will be excluded in their robots.txt file, and if a spider accesses it, the webmaster knows that the bot is a "bad bot" that does not obey robots.txt and is a good candidate for banning.
If you are on a virtual host, a lot of them will install AWstats for you or a similar stats package. Many of these stats packages will break out spider visits for you so you can see how many spiders have visited, etc., but the reports usually aren't very detailed.
anyway, just wanted to know how you can find out which spiders/bots (whatever) are searching you, and how they are indexing.
What Dan provided is perhaps secondary and the primary focus of this forum.
I learned a long time ago NEVER to assume basic items!
I have some questions that you preceed Dan's reply:
Do you know what your visitor logs are?
Are you vieweing your visitor logs?
Does your host offer visitor stats (pretty useless)?
Once you've located your visitor logs, have you noticed concentrations of successive visits by internet provdier ranges without accessing your pages full content (images and scripts included)?
With these visitor inconsistencies in hand?
Then you begin exploring some basic bot/spider identites.
Don
I do have a stat control program loaded in and can veiw all the detail of who has been veiwing site, pages, engine used....and so on. I do notice some things that look like its a search engine looking at site as they stay on for 0sec and only look at 1 page. But maybe they are not.
Thanks for your help though.
I will look into this more when i get my head around external css.
Just can't seem to get an xml menu i have in position. Have it where I want it on left ie: float:left; .... but the xml menu is sitting down about 30px below where i want it (not at the top of the div box in other words) if I use abs positioning i can get it in place, by top:0px; ....but looks totaly different when change screen settings.
Any advice? even though wrong part of forum...sorry
Geoff
ps: do any experts know where i can get hold of a script to run 4 images same size as in a slideshow but they rotate automaticaly and also fade in and out from each other?
Dynamic Drive is a good place to look for any scripts you need.
[dynamicdrive.com...]
This is one that might suit your purpose:
[dynamicdrive.com...]
use
margin-top: -30px;
