OpenWebSpider# v0.1.4

Released OpenWebSpider# v0.1.4 now with MP3s and PDFs support! New tables has been added please refer to this page to learn more: Database Structure

This is the complete CHANGELOG:

  • Mysql/NET Connector upgraded to
  • Enhanced encodings support
  • New feature: Support to META: “robots” (NOINDEX, NOFOLLOW)
  • New feature: New configuration file field: crawler_id
  • New field “crawler_id” in table “hostlist”
  • New table: crawler_act
  • New feature: Remote actions over running crawlers [Status, Play, Pause, Kill]
  • New file support: PDFs [Using PDFBox and IKVM]
  • New table: pdf
  • New file support: MP3s [Using UltraID3Lib]
  • New table: mp3
  • New feature: new command-line argument: −−pdf
  • New feature: new command-line argument: −−mp3

Go to the DOWNLOAD page

OpenWebSpider# explained with 4 video: Compile, Configure and RUN!

5 Responses

  1. alexhemp says:

    0.14 – in some random moment reported

    – Error [Wait Threads]: List has changed.
    + Killing threads…

  2. Shen139 says:

    Thanks, In certain conditions OWS could give that error.
    I fixed the bug! You can download new compiled executable here:

    I’ll release fixed code as soon as possible.

  3. Shen139 says:

    You can download here the complete package (source code + libraries + bin):

  4. alexhemp says:

    Sorry, but error still exists

    – Error [Wait Threads]: List has changed.
    + Killing threads…

    Occurs randomly, after about 1500-3000 URL’s indexed on local server.

    FreeBSD 7.0/Mono 2.4

  5. Shen139 says:

    I don’t think you are using latest version of OWS!
    Yes, I know I released a bug fixing release without changing the version number (I’ll release v0.1.5 asap);
    anyway: latest version of OWS uses different error messages for that error:
    ” – Error 1 [Wait Threads]”
    ” – Error 2 [Wait Threads]”

    The error message: “- Error [Wait Threads]” (without error number) is the message used in the previous release.

    Please download v0.1.4 from sourceforge. (maybe when you downloaded it not all servers were aligned)