Forum Moderators: open

Message Too Old, No Replies

BigSkyBot 2.06

Simple site parsing agent for niche search index

         

mcneely

9:50 pm on Nov 21, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Yes folks -- This is mine --

Parsing is extremely controlled and limited to the webmasters niche -
The only IP's you'll see this from is out at our Cascade Divide, Oregon location -- 198.37.103.58/59
Follows directives rel="nofollow/index/noindex in either page tags or properly configured .htaccess -- It alerts us when noindex/nofollow is discovered in either and we set about to dropping the page/pages/site from the queue (usually w/in 24hrs)

Ours doesn't IP hop so if you see the UA from any other IP block or range you will have my blessings on blocking it in your ,htaccess

The UA string reflects the page where you can find out about the bot and what it does -- Mozilla/5.0 (compatible; BigSkyBot/2.06; +http://www.XXXXXX.com/XXXXXX/) - No deep parsing and no data being collected/stored short of the actual listings themselves.

lucy24

1:57 am on Nov 22, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



limited to the webmasters niche
Whew. I thought it was limited to people in Montana, and that didn't seem enough of a market.