Crawl a Web site with this free ActiveX spidering component. Advanced features include caching, "avoid" patterns, and robots.txt compliance. Automatically handles HTTPS URLs. Includes safety measures for avoiding large / infinite size Web pages or infinitely growing-in-length URLs. Includes a wind-down features for crawling a maximum number of pages per site.
| License | Free |
|---|---|
| File Size | 976.5 kB |
| Version | 2.2.2 |
| Operating System | Windows XP Windows Windows 2000 |
| System Requirements |