homepage Welcome to WebmasterWorld Guest from 174.129.103.100
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Trovator
keyplyr




msg:4495305
 7:35 pm on Sep 14, 2012 (gmt 0)


robots.txt: yes, but without UA

Nova Internet ISP Network, Madrid, Spain
212.85.32.0 - 212.85.42.255

Index page only

 

dstiles




msg:4495547
 7:17 pm on Sep 15, 2012 (gmt 0)

Thanks. New to me. Blocked whole range 212.85.32.0 - 212.85.63.255

jecasc




msg:4495549
 7:23 pm on Sep 15, 2012 (gmt 0)


Thanks. New to me. Blocked whole range 212.85.32.0 - 212.85.63.255


For what reason? Are you afraid of visitors from Spain?

dstiles




msg:4496005
 9:20 pm on Sep 16, 2012 (gmt 0)

The check I made (albeit a rudimentary one) suggests the range contains servers. There are at least a couple of results in ixquick that also suggest servers.

Keyplr's report also suggests servers - or at least a bot source. Browsers seldom access robots.txt and do not browse with no UA.

Otherwise, I'm reasonably happy with Spain - one of my customers lives there. :)

jecasc




msg:4496126
 6:52 am on Sep 17, 2012 (gmt 0)

Of course it's a bot, trovator.com is a spanish search engine.
Webmasters keep complaining Google has to much power and market share. No wonder if they block every other search engine.

keyplyr




msg:4496158
 8:04 am on Sep 17, 2012 (gmt 0)

trovator.com is a spanish search engine.

More of a directory. To be included they ask you to embed a reciprocal link into one of you pages... very 20th century.

lucy24




msg:4496165
 8:19 am on Sep 17, 2012 (gmt 0)

Webmasters keep complaining Google has to much power and market share. No wonder if they block every other search engine.

Hear, hear. I haven't met trovator --keep thinking there should be an "e" at the end, wrong language-- but Seznam (Czech) seems to be very fond of my site. They've even got some kind of preview.

wilderness




msg:4496222
 11:37 am on Sep 17, 2012 (gmt 0)

Webmasters keep complaining Google has to much power and market share.


I must have missed all those complaining threads here at Webmaster World?
The Google SE bot 66\.249\.(6[4-9]|[78][0-9]|9[0-5]) doesn't upset anybody, rather their additional IP's and side-products that are annoying.

No wonder if they block every other search engine.


That last major bot to appear was the MSN stuff in 2003, the rest (non-major) are just crap.

wilderness




msg:4496225
 11:53 am on Sep 17, 2012 (gmt 0)

For what reason? Are you afraid of visitors from Spain?


It's likely that the use of "afraid" was just a lapse?

I've the entire 212 Class A denied (and many others), which includes more than just Spaniard's. I've no fear. Rather, just don't wish their traffic.

lucy24




msg:4496306
 4:32 pm on Sep 17, 2012 (gmt 0)

the rest (non-major) are just crap

Isn't that like a biologist asking why you never meet a brand-new phylum in the making? It's all subspecies.

Nobody starts out as major. Not even g###.

wilderness




msg:4496373
 7:12 pm on Sep 17, 2012 (gmt 0)

Hey lucy,
Perhaps you could provide the name of the 4th Global-major-SE behind Google, Yahoo and MSN that has appeared since 2003 and without the common knowledge of others?

The "under the theme of potential SE's" has been discussed in this forum many, many times, and before your presence here at Webmaster World.

Prior to 2003 there could have existed other major ranking SE's, however they certainly don't exist today.
Alta Vista was at one time a major bot that has literally disappeared, even though still active.

dstiles




msg:4496396
 8:12 pm on Sep 17, 2012 (gmt 0)

Jecasc - I was not aware it was an SE but since there seems to be no English explanation on the site it appears they do not want English-speaking traffic so why should I allow them access? Especially with no UA, as reported by keyplr, which means no link to their T&C.

I do not propose following it up.

lucy24




msg:4496466
 11:31 pm on Sep 17, 2012 (gmt 0)

has literally disappeared, even though still active

:: twiddling thumbs ::

"Hey, I know you. You're Alta Vista. You used to be big."
"I AM big. It's the searches that got small."

:: snrk ::

appeared since 2003 and without the common knowledge of others

People aren't letting them appear. Though there may still be a market for a "best of the rest" search engine. It's an appealing idea. Index only the sites whose owners don't know from blocking robots.

before your presence here at Webmaster World

Pulling rank now, are we :P

wilderness




msg:4496473
 11:51 pm on Sep 17, 2012 (gmt 0)

appeared since 2003 and without the common knowledge of others


People aren't letting them appear. Though there may still be a market for a "best of the rest" search engine.


Actually and likely that is not the reason at all, rather the transition of the masses to social media, and/or simply another progression of the so-called internet.

It's an appealing idea. Index only the sites whose owners don't know from blocking robots.


99.9999999% of webmasters don't give two hoots about the activities of bots on their site, nor are they even aware of the existence of raw logs.

the 0.00000001% that remain don't leave an impression influencing the activity of bots and/or harvesters.

What percentile of that 0.00000001% we here at SSID could likely fill the end of microscopic straight-pin.

wilderness




msg:4496474
 11:55 pm on Sep 17, 2012 (gmt 0)

before your presence here at Webmaster World


Pulling rank now, are we :P


Quite the contrary, I'm keen on your competitive ego and didn't wish to feel guilty when you spent useless hours searching the archives for references that are of no benefit.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved