Robots-txt Software
|
|
With Robots.txt Editor you create robot exclusion files by selecting all robots or a specific user-agent and adding documents and/or directories by entering the path names manually or by selecting them using FTP.With Log Analyzer you can get the full statistics of robots visiting the pages of your website. Keywords: robotstxt spiders site search visits spider analyzer html robots-txt log analyzer visually generate industry standard robots txt files analize webservers log |
|
|
|
Manage and track visits by robots, spiders and crawlers to your website. Identify which spiders visited, when they visited, and what pages they indexed. Direct spiders to your most important pages, and exclude them from non-public pages. Maximizes effectiveness with search engines by avoiding over/under submitting. Excellent to improve search engine placements for users of automatic submission software such as WebPosition, TopDog, AddWeb, etc. Keywords: robots-txt robot spider crawler log file log file analysis log file analyzer search engine optimization seo spider tracking referrer logs identify which spiders visited website pages indexed |
|