homepage Welcome to WebmasterWorld Guest from 54.197.111.87
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Who to create XML sitemap exclude URL parameter in Xenu's Link Sleuth
Best practices to using Xenu's Link Sleuth
rajesh magar




msg:4499603
 7:15 am on Sep 25, 2012 (gmt 0)

Hi All,

I have website which is having 3000+ article/links posted on it. And just week ago it's been redesign and now I've been wondering to create latest XML sitemap for the same.

So problem just that when I run the URL in Xenu's Link Sleuth to fetch my website links, its crawling 9000+ links with adding URL Parameter, like "?Action=" and all.

So how can I skip this URLs with addition parameter and just get crawl canonical URLs to create light weight XML sitemap.

Thanks in Advance.

 

phranque




msg:4499639
 9:32 am on Sep 25, 2012 (gmt 0)

xenu is simply going to discover the urls you link to internally, so if xenu found non-canonical url parameters you have to fix this in your web application.

rajesh magar




msg:4499664
 10:46 am on Sep 25, 2012 (gmt 0)

Ok that's fine, but is't their any way to skip this links in Xenu itself.

Like if you look at the functionality of the ScreamingFrogSEOSpider it's only crawl main pages and totally skip those all link with additional parameter in it.

Truly Appreciate you feedback, Thanks

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved