Forum Moderators: open
Do you ban it from your pages?
Regards.
[webmasterworld.com...]
[webmasterworld.com...]
Disclaimer: I am the bot's creator.
Do you ban it from your pages?
What I or any other webmaster does should NOT have any influence or correlation on what you do!
Each webmaster must determine (on their own) what is beneficial or detrimental to their own website (s).
I am the bot's creator
Could you explain where the data is used on the sites that you spider, your bot visits my server's often and takes 1,000's of pages.
Using gzip compression on your webserver is a good way to reduce traffic used by bot, if you are concerned about frequency of requests then using Crawl-Delay in robots.txt will help -- the bot will use even if its specified for other more famous bots then ours.
the pages spidered are being put to good use even if it is in the near future
The best use :)
The index right now is 44 mln, mainly pages crawled Nov'04-Feb'05, but I expect to have 10-20 times more indexed by the end of the year, and have the rest done in Q1 2006.
I will do my best to make sure that by this time next year even wilderness thought it was good to be crawled by MJ12bot ;)
where the data is used on the sites that you spider
Or where it will go in the event of your demise?
Perhaps?
"Jack"
[google.com...]
Or where it will go in the event of your demise?
It will probably go to the Silicon Heaven (tm).
The data per se has zero value (from warehousing point of view) -- its what can be done with the data (ie indexing it and making it searchable with fast and relevant results) that matters.
Or where it will go in the event of your demise?It will probably go to the Silicon Heaven (tm).
That's funny! I hope that I go there too. ;-)
That's funny! I hope that I go there too.
If you're bad, you go to Silicone Heaven - which is so bad it's good.
That's funny! I hope that I go there too. ;)
Perhaps you and majestic can arrive on the same boat ;)
I take it you are not too keen
I'm not too keen on many things when it comes to bots visiting my sites.
Some of it we must cough, gag and choke and hope they act
prudent.
Ex1: I've been getting some sparse activity from AOL Transit. Transit is the web browser that AOL folks use when they are not at home. This is unlike the other AOL blank UA and Refer fields or limitied UA of Mozilla Ver. This thing uses a standard UA.
Others we may deny outright.
For "ME"
RIPE, Major portions on APNIC, LACNIC.
Even many North American providers who fail to cooperate in enforcing their own UAG's.
Others we may deny and hope it doesn't effect our standing.
Ex: I've seen others mention the Google from 202 and it had left my sites alone until recently and they are eating 403's.
As result google has been testing if I'm spoofing pages.
One such test from a Savvis/Layered Technologies range with two different UA's (Media Partners and a standard browser) with neither reading robots and spidering anyway.)
As previously stated;
Each webamster must decide what is beneficial and detrimental to their respective websites.
"Jack" is detrimental to my websites ;)
Hopefully somebody will come along and sink the boat that he is hoping will take him to Silicon Heaven ;)
However, I still wish him the best of success and a healthy appetite for 403's ;)
Is "Jack" refering to MJ12bot?!?!
Jack is an example of another bot provided in a previous post.
Did you view the link I provided?
