Welcome to WebmasterWorld Guest from 54.145.221.99

Forum Moderators: mademetop

Message Too Old, No Replies

How to index 10,000 pages - blocked for 3 years by robots.txt

   
7:25 am on Nov 21, 2009 (gmt 0)

5+ Year Member



I have 3 old domain. Having 10,000 web pages.

Because of few reasons i used robots.txt and disallow the search engines for six months not to index the web pages. Now i have removed the disallow option.

Now i want to index 10,000 web pages by Google and remaining search engines. For this i have submitted to Google through webmaster tool and doing directory submitting, press releases,SMO

Can any one suggest me how index the pages ASAP.

4:27 pm on Nov 21, 2009 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



One good step is to create an XML Sitemap and show its location in your robots.txt file. All the search engines will respond to this.
3:37 am on Nov 23, 2009 (gmt 0)

5+ Year Member



i have create a XML sitemap and placed in robots.txt like this

Sitemap: http://www.example.com/sitemap.xml

Is it correct and any other suggestions.

[edited by: makemetop at 11:10 am (utc) on Dec. 22, 2009]
[edit reason] Changed to example.com and delinked [/edit]

6:47 am on Nov 23, 2009 (gmt 0)

5+ Year Member



For every page i have add a flash, So this the one reason those web pages are not indexing?
Can any one help me
12:46 pm on Nov 24, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



is that 10,000 pages in flash programs or 10,000 urls of documents served?
if you have replaced your textual content with flash, that will cause a problem.
you should use swfobject to provide alternate content for non-flash-enabled user agents such as SE crawlers.
8:56 am on Dec 22, 2009 (gmt 0)

5+ Year Member



Hi,

the location of

files

http://www.example.com/category/subcagtegory/abac.html

My website is located like this. How to index the pages. Is there any tips for this?

[edited by: makemetop at 11:08 am (utc) on Dec. 22, 2009]
[edit reason] Changed to example.com and delinked [/edit]

11:12 am on Dec 22, 2009 (gmt 0)



Make sure all directories and subdirectories are linked to from your sitemap. As long as all sections of your site are linked to, they should be indexed OK.

Of course, there is a big difference in being indexed and actually ranking for anything - but it is an important first step.

12:20 pm on Dec 22, 2009 (gmt 0)

5+ Year Member



Hi,

Means i have to place properly in sitemap.html file or i have to include in sitemap.xml file.

Can you explain me briefly.

1:32 pm on Dec 22, 2009 (gmt 0)



I would do both, if you have both.

Certainly you should at least have an html version for all spiders.

2:42 pm on Dec 22, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Not only for search engines but for usability too - Not everyone has flash enabled or people with a reading disability would have trouble enlarging the text in a flash only website.
5:02 am on Dec 23, 2009 (gmt 0)

5+ Year Member



i have content website and using flash in header of the page in all 10k pages. Is this will effect for indexing my pages.

If so what to do? How to index those pages

7:49 am on Dec 23, 2009 (gmt 0)

5+ Year Member



For indexing fast, after creating sitemap.xml submit it to webmaster tools. By doing this, not only indexing will be fast but also if any issue or errors will be occur in sitemap.xml will be report to you by the tool. Also try to get some links from good and fast indexing site because it also help much to crawl your site by search engine crawlers.
7:57 am on Dec 23, 2009 (gmt 0)

5+ Year Member



Hi Suresh,

As i had previously mentioned in my first post

"I have 3 old domain. Having 10,000 web pages.

Because of few reasons i used robots.txt and disallow the search engines for six months not to index the web pages. Now i have removed the disallow option.

Now i want to index 10,000 web pages by Google and remaining search engines. For this i have submitted to Google through webmaster tool and doing directory submitting, press releases,SMO

Can any one suggest me how index the pages ASAP."

11:57 am on Dec 29, 2009 (gmt 0)

5+ Year Member



can any one suggest me to index 10,000 pages
8:06 am on Dec 30, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



is GWT showing any problem related to crawling your site?
have you looked at your access logs to see how googlebot is crawling your urls?
do you have inbound links from urls that are already indexed?
10:32 am on Dec 30, 2009 (gmt 0)

5+ Year Member



In GWT is not showing any error. yes i have inbound links total 180 and internal links too.