homepage Welcome to WebmasterWorld Guest from 54.211.7.174
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
FTP.Domain.com Ranking Higher Than www.Domain.com
Why is my FTP site ranking higher than my www.?
mpeters742




msg:4624593
 7:59 pm on Nov 19, 2013 (gmt 0)

Hello All,

Our site has had a rough couple of weeks all starting when IT inadvertently blocked Google from crawling our site in robots.txt. So naturally organic traffic dropped of the face of the planet. We got robots.txt fixed and Google can now crawl our site but of course organic traffic isn't returning as quickly as management would like.

I've gone into Webmaster Tools and fetched as Google all our important pages and resubmitted them to the index along with any associated pages. No fails there so you'd think we're on the road to recovery but even after 2 weeks some of our most popular pages haven't been indexed yet.

Even stranger we have ftp.domainname.com ranking in place of www.domainname.com. Now we've blocked Google from hitting our FTP site and currently 301 redirect any ftp.domainname.com calls to www.domainname.com and when I do a "site:www.domainname.com/pageiwant" search on Google and it is nowhere to be found but the ftp version is.

Now I Fetched as Googlebot this URL 2 weeks ago and submitted it to index after the successful fetch. Is this a case where we just need to wait for Google or is there something else I could be doing to speed up the process?

Thanks for any insights,
Mike

 

phranque




msg:4624690
 6:47 am on Nov 20, 2013 (gmt 0)

welcome to WebmasterWorld, mpeters742!


how are you blocking googlebot?
if you block googlebot from crawling the ftp site it won't see the 301.

mpeters742




msg:4624815
 3:21 pm on Nov 20, 2013 (gmt 0)

Upon double checking with my IT department it seems I misspoke. We are actually just 301 redirecting, not blocking FTP.

mpeters742




msg:4624890
 8:17 pm on Nov 20, 2013 (gmt 0)

Jumping even deeper into this my IT folks tell me that Google didn't index our FTP they indexed http/https protocols because the host name was mistakenly available on ports 80/443

http://ftp.domain.com/robots.txt
https://ftp.domain.com/robots.txt

This makes zero sense to me :(

[edited by: phranque at 5:11 pm (utc) on Nov 21, 2013]
[edit reason] unlinked url [/edit]

phranque




msg:4625141
 5:12 pm on Nov 21, 2013 (gmt 0)

forget what IT tells you.
go to GWT, "fetch as googlebot" and request https://ftp.domain.com/robots.txt

what response does googlebot see?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved