OpenWebSpider http://www.openwebspider.org Open Source Web Spider Thu, 07 May 2009 10:24:14 +0000 http://wordpress.org/?v=2.9.2 en hourly 1 OpenWebSpider# v0.1.4 http://www.openwebspider.org/2009/05/07/openwebspider-v014/ http://www.openwebspider.org/2009/05/07/openwebspider-v014/#comments Thu, 07 May 2009 09:02:56 +0000 Shen139 http://www.openwebspider.org/?p=115 Released OpenWebSpider# v0.1.4 now with MP3s and PDFs support! New tables has been added please refer to this page to learn more: Database Structure

This is the complete CHANGELOG:

  • Mysql/NET Connector upgraded to 5.2.5.0
  • Enhanced encodings support
  • New feature: Support to META: “robots” (NOINDEX, NOFOLLOW)
  • New feature: New configuration file field: crawler_id
  • New field “crawler_id” in table “hostlist”
  • New table: crawler_act
  • New feature: Remote actions over running crawlers [Status, Play, Pause, Kill]
  • New file support: PDFs [Using PDFBox and IKVM]
  • New table: pdf
  • New file support: MP3s [Using UltraID3Lib]
  • New table: mp3
  • New feature: new command-line argument: −−pdf
  • New feature: new command-line argument: −−mp3

Go to the DOWNLOAD page

OpenWebSpider# explained with 4 video: Compile, Configure and RUN!

]]> http://www.openwebspider.org/2009/05/07/openwebspider-v014/feed/ 5 Mono 2.4 has been released http://www.openwebspider.org/2009/04/11/mono-24-has-been-released/ http://www.openwebspider.org/2009/04/11/mono-24-has-been-released/#comments Sat, 11 Apr 2009 09:38:11 +0000 Shen139 http://www.openwebspider.org/?p=106 mono_24

Mono 2.4 has been released! The Mono Project aims to make developers productive and happy: Mono 2.4 is our gift to the world. Sponsored by Novell, the Mono open source project has an active and enthusiastic contributing community and is positioned to become the leading choice for development of Linux applications.

Mono Feature Highlights

  • Multi-Platform
    Runs on Linux, OS X, BSD, and Microsoft Windows, including x86, x86-64, ARM, s390, PowerPC and much more
  • Multi-Language
    Develop in C# 3.0 (including LINQ), VB 8, Java, Python, Ruby (http://www.ironruby.net/), Eiffel (http://www.eiffel.com/), F# (http://research.microsoft.com/fsharp/), Oxygene (http://remobjects.com/oxygene), and more
  • Binary Compatible
    Built on an implementation of the ECMA’s Common Language Infrastructure and C#
  • Microsoft Compatible API
    Run ASP.NET, ADO.NET, and Windows.Forms 2.0 applications without recompilation
  • Open Source, Free Software
    Mono’s runtime, compilers, and libraries are distributed under OSI approved licenses and are available for dual-licensing
  • Comprehensive Technology Coverage
    Bindings and managed implementations of many popular libraries and protocols

download Mono

]]> http://www.openwebspider.org/2009/04/11/mono-24-has-been-released/feed/ 0 OpenWebSpider# v0.1.3 http://www.openwebspider.org/2008/11/05/openwebspider-v013/ http://www.openwebspider.org/2008/11/05/openwebspider-v013/#comments Wed, 05 Nov 2008 10:40:38 +0000 Shen139 http://www.openwebspider.org/?p=96 Released OpenWebSpider v0.1.3

CHANGELOG:

  • New feature: CRAWLER NAME and CRAWLER VERSION used in the User-Agent string in HTTP Requests
  • New feature: New configuration file field: sql_hostlist_where
  • New feature: new command-line argument: –keep-dup
  • BUG: fixed the regex used to extract URLs from <BASE>
  • BUG: fixed in the function that extracts URLs
  • BUG: fixed a bug in page.cs::normalizePage()
  • BUG: fixed minor bugs
  • BUG: fixed a bug in robots.txt’s parser
  • BUG: fixed a bug in page-rels handler

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1

]]> http://www.openwebspider.org/2008/11/05/openwebspider-v013/feed/ 2 Mono 2.0 has been released http://www.openwebspider.org/2008/10/07/mono-20-has-been-released/ http://www.openwebspider.org/2008/10/07/mono-20-has-been-released/#comments Tue, 07 Oct 2008 09:38:17 +0000 Shen139 http://www.openwebspider.org/?p=77 The Mono Project aims to make developers productive and happy: Mono 2.0 is our gift to the world.
Sponsored by Novell (http://www.novell.com), the Mono open source project has an active and enthusiastic contributing community and is positioned to become the leading choice for development of Linux applications.

Feature Highlights

Multi-Platform
Runs on Linux, OS X, BSD, and Microsoft Windows, including x86, x86-64, ARM, s390, PowerPC and much more
Multi-Language
Develop in C# 3.0 (including LINQ), VB 8, Java, Python, Ruby (http://www.ironruby.net/), Eiffel (http://www.eiffel.com/), F# (http://research.microsoft.com/fsharp/), Oxygene (http://remobjects.com/oxygene), and more
Based on ECMA Standards
Built on an implementation of the ECMA Common Language Infrastructure and C#
Microsoft Compatible API
Run ASP.NET, ADO.NET, and Windows.Forms 2.0 applications without recompilation
Open Source, Free Software
Mono’s runtime, compilers, and libraries are distributed under OSI approved licenses and are available for dual-licensing
Comprehensive Technology Coverage
Bindings and managed implementations of many popular libraries and protocols
]]> http://www.openwebspider.org/2008/10/07/mono-20-has-been-released/feed/ 0 OpenWebSpider# v0.1.2 http://www.openwebspider.org/2008/09/09/openwebspider-v012/ http://www.openwebspider.org/2008/09/09/openwebspider-v012/#comments Tue, 09 Sep 2008 10:01:48 +0000 Shen139 http://www.openwebspider.org/?p=62 Released OpenWebSpider v0.1.2

CHANGELOG:

  • BUG: fixed the regex used to extract URLs from (I)FRAME
  • New feature: OpenWebSpider# can index images (new table: images)
  • New feature: new command-line argument: −−images
  • Improved Stress-test facility: now OpenWebSpider# doesn’t require a configuration file and a MySQL Server and it doesn’t check robots.txt (in stress-test mode)
  • Timeout in execution of SQL queries set to 120 seconds (2 minutes)
  • New feature: new configuration file fields: CRAWLER NAME and CRAWLER VERSION
  • New feature: CRAWLER NAME used over robots.txt

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1

]]> http://www.openwebspider.org/2008/09/09/openwebspider-v012/feed/ 24 OpenWebSpider# v0.1.1 http://www.openwebspider.org/2008/08/21/openwebspider-v011/ http://www.openwebspider.org/2008/08/21/openwebspider-v011/#comments Thu, 21 Aug 2008 09:36:21 +0000 Shen139 http://www.openwebspider.org/?p=51 Released OpenWebSpider v0.1.1

CHANGELOG:

  • New feature: new command-line argument: −−req−timeout
  • New feature: new command-line argument: −−stress−test
  • BUG: fixed a bug in http.cs::getURL()

[New features here: OpenWebSpider# v0.1 Command Line Arguments/Usage]

[ Read more about: Why C#? Why .NET Framework? ]

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1

]]> http://www.openwebspider.org/2008/08/21/openwebspider-v011/feed/ 22 New Video: OpenWebSpider# Command Line Arguments/Basic Usage + Example under Linux http://www.openwebspider.org/2008/07/31/new-video-openwebspider-command-line-argumentsbasic-usage-example-under-linux/ http://www.openwebspider.org/2008/07/31/new-video-openwebspider-command-line-argumentsbasic-usage-example-under-linux/#comments Thu, 31 Jul 2008 13:24:35 +0000 Shen139 http://www.openwebspider.org/?p=50 openwebspider# Usage Example

OpenWebSpider# Command Line Arguments/Basic Usage + Example under Linux

Click here to watch the Video!





]]> http://www.openwebspider.org/2008/07/31/new-video-openwebspider-command-line-argumentsbasic-usage-example-under-linux/feed/ 0 OpenWebSpider# v0.1 http://www.openwebspider.org/2008/07/29/openwebspider-v01/ http://www.openwebspider.org/2008/07/29/openwebspider-v01/#comments Tue, 29 Jul 2008 21:41:22 +0000 Shen139 http://www.openwebspider.org/?p=37 Released the first public version of OpenWebSpider entirely written in C#
[ Read more about: Why C#? Why .NET Framework? ]

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1

]]> http://www.openwebspider.org/2008/07/29/openwebspider-v01/feed/ 19 Create Database And Tables for OpenWebSpider v0.7 Video http://www.openwebspider.org/2008/07/21/create-database-and-tables-for-openwebspider-v07-video/ http://www.openwebspider.org/2008/07/21/create-database-and-tables-for-openwebspider-v07-video/#comments Mon, 21 Jul 2008 09:21:06 +0000 Shen139 http://www.openwebspider.org/?p=11 Create Database And Tables For Openwebspider
Video: Create Database And Tables for OpenWebSpider v0.7
Click here to see the video!

]]> http://www.openwebspider.org/2008/07/21/create-database-and-tables-for-openwebspider-v07-video/feed/ 0 OpenWebSpider v0.7 http://www.openwebspider.org/2008/07/21/openwebspider-v07/ http://www.openwebspider.org/2008/07/21/openwebspider-v07/#comments Mon, 21 Jul 2008 09:19:39 +0000 Shen139 http://www.openwebspider.org/?p=10 Released OpenWebSpider v0.7 Source Code + Win32 Binary [Download]

]]> http://www.openwebspider.org/2008/07/21/openwebspider-v07/feed/ 0