Ever wish you could just open up a program and have it generate an index of all links on your site? You need this program, you need the search engines to have this precious file called a sitemap.xml. It’s so important you have this file that all search engines check it when they crawl your site.
How to install
1. Download all the files and unzip them in a browsable directory
2. Edit config.inc and change the example.com in the URL_seed and DOMAIN to whatever site you wish to crawl and generate data for.
3. Browse to the directory you unzipped in, then wait. The program will do it’s run.
It’s to be noted that if you close the window while it’s running it will interrupt the process, but here is the good news. Just reopen it, it will find where it left off and pickup again. This is because what this crawler really does.
This actual program is a search engine crawler as well. It’s reading every file and parsing out the words and creating a matrix of words and frequency. Just like Google it will create keyword densities. If this was coupled with the search engine we created you would have a fully function search engine and crawler.
If you are interested in the search engine data and portion, please contact us and we could certainly help you. We also plan in the future to release a full package of this.
Search engines 
