crawler-commons is a set of reusable Java components that implement functionality common to any web crawler. These components benefit from collaboration among various existing web crawler projects, and reduce duplication of effort.
We are glad to announce the 0.6 release of Crawler Commons. See the CHANGES.txt file included with the release for a full list of details.
We suggest all users to upgrade to this version. Details of how to do so can be found on Maven Central. Please note that the groupId has changed to com.github.crawler-commons.
The Java documentation can be found here.
The crawler-commons project is now being hosted at GitHub, due to the demise of Google code hosting.
We are glad to announce the 0.5 release of Crawler Commons. This release mainly improves Sitemap parsing as well as an upgrade to Apache Tika 1.6.
See the CHANGES.txt file included with the release for a full list of details. Additionally the Java documentation can be found here.
We suggest all users to upgrade to this version. The Crawler Commons project artifacts are released as Maven artifacts and can be found at Maven Central.
We are glad to announce the 0.4 release of Crawler Commons. Amongst other improvements, this release includes support for Googlebot-compatible regular expressions in URL specifications, further imprvements to robots.txt parsing and an upgrade of httpclient to v4.2.6.
See the CHANGES.txt file included with the release for a full list of details.
We suggest all users to upgrade to this version. Details of how to do so can be found on Maven Central.
This release improves robots.txt and sitemap parsing support, updates Tika to the latest released version (1.4), and removes some left-over cruft from the pre-Maven build setup.
See the CHANGES.txt file included with the release for a full list of details.
Similar to the previous note about Nutch 2.2, there's now a version of Nutch in the 1.x tree that also uses crawler-commons. See Apache Nutch v1.7 Released for more details.
See Apache Nutch v2.2 Released for more details.
This release improves robots.txt and sitemap parsing support.
See the CHANGES.txt file included with the release for a full list of details.
There is a mailing list on Google Groups.
If you find an issue, please file a report here