Chilkat Spider ActiveX

Crawl a Web site with this free ActiveX spidering component. Advanced features include caching, "avoid" patterns, and robots.txt compliance. Automatically handles HTTPS URLs. Includes safety measures for avoiding large / infinite size Web pages or infinitely growing-in-length URLs. Includes a wind-down features for crawling a maximum number of pages per site.
Price USD 0
License Free
File Size 976.5 kB
Version 2.2.2
Operating System Windows, Windows 2000, Windows XP
System Requirements None