Crawl a website with ActiveX Spidering component. Advanced features include caching, "avoid" patterns, and robots.txt compliance.
Spider ActiveX Component for Crawling the Web:
-Cache pages so future crawls can fetch from cache;
-Robots.txt compliant;
-Fetch the HTML content of each page crawled;
-Able to crawl HTTPS pages;
-Define "avoid" patterns to avoid URLs matching specific wildcard patterns;
-Define "avoid" patterns for avoiding matching outbound links.
Comments