Web Crawler

The web crawler leverages your website sitemap to automatically crawl and index your pages.

1. Input domain and extract pages

2. Select folders of interest

3. Blacklist certain URL patterns

4. Name crawl and launch