OpenWebSpider » openwebspider http://www.openwebspider.org Open Source Web Spider Mon, 18 May 2009 09:59:46 +0000 en hourly 1 http://wordpress.org/?v=3.2.1 OpenWebSpider# v0.1.3 http://www.openwebspider.org/2008/11/05/openwebspider-v013/ http://www.openwebspider.org/2008/11/05/openwebspider-v013/#comments Wed, 05 Nov 2008 10:40:38 +0000 Shen139 http://www.openwebspider.org/?p=96 Released OpenWebSpider v0.1.3

CHANGELOG:

  • New feature: CRAWLER NAME and CRAWLER VERSION used in the User-Agent string in HTTP Requests
  • New feature: New configuration file field: sql_hostlist_where
  • New feature: new command-line argument: –keep-dup
  • BUG: fixed the regex used to extract URLs from <BASE>
  • BUG: fixed in the function that extracts URLs
  • BUG: fixed a bug in page.cs::normalizePage()
  • BUG: fixed minor bugs
  • BUG: fixed a bug in robots.txt’s parser
  • BUG: fixed a bug in page-rels handler

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1

]]> http://www.openwebspider.org/2008/11/05/openwebspider-v013/feed/ 2 OpenWebSpider# v0.1.1 http://www.openwebspider.org/2008/08/21/openwebspider-v011/ http://www.openwebspider.org/2008/08/21/openwebspider-v011/#comments Thu, 21 Aug 2008 09:36:21 +0000 Shen139 http://www.openwebspider.org/?p=51 Released OpenWebSpider v0.1.1

CHANGELOG:

  • New feature: new command-line argument: −−req−timeout
  • New feature: new command-line argument: −−stress−test
  • BUG: fixed a bug in http.cs::getURL()

[New features here: OpenWebSpider# v0.1 Command Line Arguments/Usage]

[ Read more about: Why C#? Why .NET Framework? ]

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1

]]> http://www.openwebspider.org/2008/08/21/openwebspider-v011/feed/ 22 New Video: OpenWebSpider# Command Line Arguments/Basic Usage + Example under Linux http://www.openwebspider.org/2008/07/31/new-video-openwebspider-command-line-argumentsbasic-usage-example-under-linux/ http://www.openwebspider.org/2008/07/31/new-video-openwebspider-command-line-argumentsbasic-usage-example-under-linux/#comments Thu, 31 Jul 2008 13:24:35 +0000 Shen139 http://www.openwebspider.org/?p=50 openwebspider# Usage Example

OpenWebSpider# Command Line Arguments/Basic Usage + Example under Linux

Click here to watch the Video!





]]> http://www.openwebspider.org/2008/07/31/new-video-openwebspider-command-line-argumentsbasic-usage-example-under-linux/feed/ 0 OpenWebSpider# v0.1 http://www.openwebspider.org/2008/07/29/openwebspider-v01/ http://www.openwebspider.org/2008/07/29/openwebspider-v01/#comments Tue, 29 Jul 2008 21:41:22 +0000 Shen139 http://www.openwebspider.org/?p=37 Released the first public version of OpenWebSpider entirely written in C#
[ Read more about: Why C#? Why .NET Framework? ]

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1

]]> http://www.openwebspider.org/2008/07/29/openwebspider-v01/feed/ 19 Create Database And Tables for OpenWebSpider v0.7 Video http://www.openwebspider.org/2008/07/21/create-database-and-tables-for-openwebspider-v07-video/ http://www.openwebspider.org/2008/07/21/create-database-and-tables-for-openwebspider-v07-video/#comments Mon, 21 Jul 2008 09:21:06 +0000 Shen139 http://www.openwebspider.org/?p=11 Create Database And Tables For Openwebspider
Video: Create Database And Tables for OpenWebSpider v0.7
Click here to see the video!

]]> http://www.openwebspider.org/2008/07/21/create-database-and-tables-for-openwebspider-v07-video/feed/ 0 OpenWebSpider v0.7 http://www.openwebspider.org/2008/07/21/openwebspider-v07/ http://www.openwebspider.org/2008/07/21/openwebspider-v07/#comments Mon, 21 Jul 2008 09:19:39 +0000 Shen139 http://www.openwebspider.org/?p=10 Released OpenWebSpider v0.7 Source Code + Win32 Binary [Download]

]]> http://www.openwebspider.org/2008/07/21/openwebspider-v07/feed/ 0 OpenWebSpider v0.7 Video http://www.openwebspider.org/2008/07/21/openwebspider-v07-video/ http://www.openwebspider.org/2008/07/21/openwebspider-v07-video/#comments Mon, 21 Jul 2008 09:18:03 +0000 Shen139 http://www.openwebspider.org/?p=9 Compile And Run Openwebspider v0.7 under Linux
Video: Compile And Run Openwebspider v0.7 under Linux
Click here to see the video!

]]> http://www.openwebspider.org/2008/07/21/openwebspider-v07-video/feed/ 0 OpenWebSpider v0.6.1 http://www.openwebspider.org/2008/07/21/openwebspider-v061/ http://www.openwebspider.org/2008/07/21/openwebspider-v061/#comments Mon, 21 Jul 2008 09:17:01 +0000 Shen139 http://www.openwebspider.org/?p=8 Released OpenWebSpider v0.6.1 Source Code + Win32 Binary [Download]

]]> http://www.openwebspider.org/2008/07/21/openwebspider-v061/feed/ 0