OpenWebSpider

Released OpenWebSpider v0.1.3

CHANGELOG:

• New feature: CRAWLER NAME and CRAWLER VERSION used in the User-Agent string in HTTP Requests
• New feature: New configuration file field: sql_hostlist_where
• New feature: new command-line argument: –keep-dup
• BUG: fixed the regex used to extract URLs from <BASE>
• BUG: fixed in the function that extracts URLs
• BUG: fixed a bug in page.cs::normalizePage()
• BUG: fixed minor bugs
• BUG: fixed a bug in robots.txt’s parser
• BUG: fixed a bug in page-rels handler

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1

Released OpenWebSpider v0.1.2

CHANGELOG:

• BUG: fixed the regex used to extract URLs from (I)FRAME
• New feature: OpenWebSpider# can index images (new table: images)
• New feature: new command-line argument: −−images
• Improved Stress-test facility: now OpenWebSpider# doesn’t require a configuration file and a MySQL Server and it doesn’t check robots.txt (in stress-test mode)
• Timeout in execution of SQL queries set to 120 seconds (2 minutes)
• New feature: new configuration file fields: CRAWLER NAME and CRAWLER VERSION
• New feature: CRAWLER NAME used over robots.txt

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1