If you want to see a working search that is indexed by sphinx, and spidered by ows check out http://www.hfvs.net/ Also the database that i use for sites to spider can be found at: http://cyberdawn.w3dt.net/ it contains about 90 million tld’s.

You have my email, when you want you can send me your work and I’ll publish it!

I have no problem if you want to add the source code front end to your project .

When i get time i want to create some ready made smarty templates for ows front end putting all this together would make a great free search engine project.

Regard

Darren

PS Big Thanks to you Shen for creating OWS with out your skills and giving up your time we could not follow our online adventures!

I think that http://www.linuxhostuk.co.uk/searchdemo really rocks it’s fast and cool!
I’m interested in your front-end, I’ll be very happy to publish it here!

Please contact me @ shen139 (at) openwebspider [.] org

I forgot to say i had new front end made for ows which uses sphinx it makes a powerfull alternative to the standard search its free i was trying to give back i was wondering if you would like to take a look and maybe add to ows site as an add on to ows users the search demo for ows & sphinx is here http://www.linuxhostuk.co.uk/searchdemo let me know your thoughts.

Regards

Darren

I thought it was me going mad many thanks i look forward to the fix

regards

darren

I’ve tested the crlawler (a version with little modification respect the public one) over: “download.opensuse.org” (where there is a lot of un-handled content-type (ISO, torrent, …) )

command line arguments:
–index http://download.opensuse.org/ –threads 6 –add-external -s

–[
...
T[0] Downloading… [ http://download.opensuse.org:80/distribution/11.0/iso/dvd/openSUSE-11.0-DVD-i386.iso ] [Depth Level: 1]
T[0] Downloaded 0 Kb (0 bytes) in 4609 ms
T[0] HTTP Status Code: 200 -][- Content-Type: application/x-iso9660-image
T[0] Indexing…NOT INDEXED [0 ms ]

T[2] Downloading… [ http://download.opensuse.org:80/distribution/11.0/iso/cd/openSUSE-11.0-GNOME-LiveCD-i386.iso ] [Depth Level: 1]
T[2] Downloaded 0 Kb (0 bytes) in 53015 ms
T[2] HTTP Status Code: 200 -][- Content-Type: application/octet-stream
T[2] Indexing…NOT INDEXED [0 ms ]

T[2] Downloading… [ http://download.opensuse.org:80/distribution/11.0/iso/torrent/openSUSE-11.0-DVD-i386.torrent ] [Depth Level: 1]
T[2] Downloaded 0 Kb (0 bytes) in 56609 ms
T[2] HTTP Status Code: 200 -][- Content-Type: application/x-bittorrent
T[2] Indexing…NOT INDEXED [0 ms ]

T[2] Downloading… [ http://download.opensuse.org:80/distribution/11.0/iso/dvd/openSUSE-11.0-DVD-x86_64.iso.metalink ] [Depth Level: 2]
T[2] Downloaded 0 Kb (0 bytes) in 953 ms
T[2] HTTP Status Code: 200 -][- Content-Type: application/metalink+xml; charset=UTF-8
T[2] Indexing…NOT INDEXED [0 ms ]
…
]–
everything seems to work well!!!

(few minutes later)

oh s..t!
I’ve tested the same website with OpenWebSpiderCS v0.1 and it doesn’t work well
I fix the code, I add 2 new features and I publish it!!!

Thanks for the suggestion!

Going back to my post when ows see’s zip,file etc it it gives me ‘ HTTP Status Code: 0 -] Error: Response received from server was null ‘ then stops crawling and every link returns the same error.

I have reduced the threads to only 6 and also set crawl delay but still will not index after hitting zip file or ISO file etc

This is my commands

–index http://www.fedora.org –threads 6 –add-external –crawl-delay 1

may be you can point me in right direction in how to find what maybe causing this error.

Regards

Darren

Another problem could be that the Web Server is under load and can’t serve pages within 60 seconds!
I can suggest you to:
- use a crawl delay (–crawl-delay )
the number of seconds between the download of a page and the next one
- use a lower number of threads

When ows hit files that has a content-type un-handled it doesn’t download them at all, so I think that your case is only a coincidence.

“openwebspider.vcproj” is the project file for OWS v0.7

MonoDevelop is an IDE for building .NET projects under Linux!