Web 2 Market maintains an updated SpiderAgents.txt file that works with both the CFMX and ASP.net versions of AbleCommerce. SpiderAgents.txt was an update to AbleCommerce in build 1340 of the CFMX version and was provided as a patch to build 1236 of the ASP version. AbleCommerce has provided a basic SpiderAgents.txt file included with subsequent builds of both versions. With this file search engines can index the store pages without including any session information in the indexed URLs.
Web 2 Market's SpiderAgents.txt file is a more current list of spiders than the default file included with AbleCommerce releases. We use sophisticated tracking and logging code combined with manual inspection of all the listed spider agents to ensure that this list is the most complete list available.
New: 6/30/2009 - Now identifying 5680 Spider Agents, including the latest from Google, Yahoo, and MSN!
For more information about User Agents, Spiders, Robots, and Crawlers, visit the sites below:
File updated as of June of 2009!