homepage Welcome to WebmasterWorld Guest from 54.166.122.65
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Do Search Engine Robots Crawl Sites with no Domain Name?
montman

5+ Year Member



 
Msg#: 4036537 posted 10:52 pm on Dec 3, 2009 (gmt 0)

Over the course of about a week, and up until last Saturday evening, I was testing web pages on my ISP free hosting server. Sometimes I left the pages up for a few hours at a time before taking them down again. I was using this just for tests, and the pages were on my ISP without a domain name. I have since moved the pages to a hosting company with a domain name

Unfortunately, I was not aware of the ability to keep most robots from crawling the pages with a robot.txt file.

I am concerned that search engines might have crawled the pages. However, I'm hoping that because the pages did not have a domain name, and maybe even because the were on an ISP server, that it is unlikely they were crawled, and if crawled, unlikely they will be indexed. Are these good reasons why they probably wouldn't be crawled or indexed?

 

LifeinAsia

WebmasterWorld Administrator lifeinasia us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4036537 posted 11:04 pm on Dec 3, 2009 (gmt 0)

If the search engines found 'em, they probably started spidering 'em.

montman

5+ Year Member



 
Msg#: 4036537 posted 11:16 pm on Dec 3, 2009 (gmt 0)

Thank you for your response.

What I'm wondering is, is it unlikely that the search engines found them because the were on an ISP without a domain name?

willybfriendly

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4036537 posted 11:34 pm on Dec 3, 2009 (gmt 0)

If you have a G Toolbar they were probably found...

montman

5+ Year Member



 
Msg#: 4036537 posted 11:40 pm on Dec 3, 2009 (gmt 0)

I'm sorry, I don't understand. I assume a G toolbar is a Google toolbar. I have a Google toolbar in one of my browsers, but surely this wouldn't effect the likelihood of the web pages I was testing being found--or could it somehow?

LifeinAsia

WebmasterWorld Administrator lifeinasia us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4036537 posted 11:55 pm on Dec 3, 2009 (gmt 0)

Definitely. G toolbar tracks where you've been and sends the info to Google. Similarly if you had AdSense code on any of the pages, most likely you were visited by the AdSense bot as well.

montman

5+ Year Member



 
Msg#: 4036537 posted 12:08 am on Dec 4, 2009 (gmt 0)

I see. But now I'm wondering: I have the G toolbar in Firefox, but I have not used the G toolbar for searches. I always use the regular Google search field in the top right of my browser instead. So, would the toolbar track my searches and send the information to Google if I didn't use the G toollbar for searches?

montman

5+ Year Member



 
Msg#: 4036537 posted 12:19 am on Dec 4, 2009 (gmt 0)

I was off track on my last post. I guess its not my searches that G toolbar tracks, but the pages I visit, and just that fact that I have the G toolbar means that my visits are sent to Google, even if I don't actually use the G toolbar. Is that right?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved