Verity Spider

The Verity Spider (vspider.exe) is a hidden gem that is found in the OEM verity K2 engine that ships with ColdFusion 5. It's well documented and relatively easy to get going. The spider can crawl across web pages and file collections and has a whole plethora of options.

The Problem

The collection generated by vspider does not return all the fields necessary to build a search interface when queried using CFSEARCH. This is very frustrating because everything works perfectly except you get no keys in the result set... that is you can search the index but it does not return the actual URL of the destination that was spidered!

Getting the Spider Collection to Work

Vspider.exe is a utility that creates an External Verity Collection. "External" simply means that the collection has not been created through the Verity/ColdFusion interface, and consequently ColdFusion doesn't have a complete understanding of what is in the collection.

All we have to do is modify the Verity collection configuration files to map relevant information to the fields that ColdFusion expects to find in the collection. This would really be quite easy if there was some sort of documentation in this area... but alas.