OpenWebSpider

Mono 2.4 has been released! The Mono Project aims to make developers productive and happy: Mono 2.4 is our gift to the world. Sponsored by Novell, the Mono open source project has an active and enthusiastic contributing community and is positioned to become the leading choice for development of Linux applications.

Continue Reading »

Released OpenWebSpider v0.1.3

CHANGELOG:

• New feature: CRAWLER NAME and CRAWLER VERSION used in the User-Agent string in HTTP Requests
• New feature: New configuration file field: sql_hostlist_where
• New feature: new command-line argument: –keep-dup
• BUG: fixed the regex used to extract URLs from <BASE>
• BUG: fixed in the function that extracts URLs
• BUG: fixed a bug in page.cs::normalizePage()
• BUG: fixed minor bugs
• BUG: fixed a bug in robots.txt’s parser
• BUG: fixed a bug in page-rels handler

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1

The Mono Project aims to make developers productive and happy: Mono 2.0 is our gift to the world.
Sponsored by Novell (http://www.novell.com), the Mono open source project has an active and enthusiastic contributing community and is positioned to become the leading choice for development of Linux applications.

Feature Highlights

Multi-Platform

Runs on Linux, OS X, BSD, and Microsoft Windows, including x86, x86-64, ARM, s390, PowerPC and much more

Multi-Language

Develop in C# 3.0 (including LINQ), VB 8, Java, Python, Ruby (http://www.ironruby.net/), Eiffel (http://www.eiffel.com/), F# (http://research.microsoft.com/fsharp/), Oxygene (http://remobjects.com/oxygene), and more

Based on ECMA Standards

Built on an implementation of the ECMA Common Language Infrastructure and C#

Microsoft Compatible API

Run ASP.NET, ADO.NET, and Windows.Forms 2.0 applications without recompilation

Open Source, Free Software

Mono’s runtime, compilers, and libraries are distributed under OSI approved licenses and are available for dual-licensing

Comprehensive Technology Coverage

Bindings and managed implementations of many popular libraries and protocols

Released OpenWebSpider v0.1.1

CHANGELOG:

• New feature: new command-line argument: −−req−timeout
• New feature: new command-line argument: −−stress−test
• BUG: fixed a bug in http.cs::getURL()

[New features here: OpenWebSpider# v0.1 Command Line Arguments/Usage]

[ Read more about: Why C#? Why .NET Framework? ]

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1

Released the first public version of OpenWebSpider entirely written in C#
[ Read more about: Why C#? Why .NET Framework? ]

Source code and binary are available in the package: Download

Documentation of OpenWebSpider# v0.1

Released OpenWebSpider v0.7 Source Code + Win32 Binary [Download]

Released OpenWebSpider v0.6.1 Source Code + Win32 Binary [Download]