OpenWebSpider# v0.1.3

Shen139 — Wed, 05 Nov 2008 10:40:38 +0000

Released OpenWebSpider v0.1.3

CHANGELOG:

New feature: CRAWLER NAME and CRAWLER VERSION used in the User-Agent string in HTTP Requests
New feature: New configuration file field: sql_hostlist_where
New feature: new command-line argument: –keep-dup
BUG: fixed the regex used to extract URLs from
BUG: fixed in the function that extracts URLs
BUG: fixed a bug in page.cs::normalizePage()
BUG: fixed minor bugs
BUG: fixed a bug in robots.txt’s parser
BUG: fixed a bug in page-rels handler

Source code and binary are available in the package: Download

OpenWebSpider# v0.1.2

Shen139 — Tue, 09 Sep 2008 10:01:48 +0000

Released OpenWebSpider v0.1.2

CHANGELOG:

BUG: fixed the regex used to extract URLs from (I)FRAME
New feature: OpenWebSpider# can index images (new table: images)
New feature: new command-line argument: −−images
Improved Stress-test facility: now OpenWebSpider# doesn’t require a configuration file and a MySQL Server and it doesn’t check robots.txt (in stress-test mode)
Timeout in execution of SQL queries set to 120 seconds (2 minutes)
New feature: new configuration file fields: CRAWLER NAME and CRAWLER VERSION
New feature: CRAWLER NAME used over robots.txt

Source code and binary are available in the package: Download