Comments on: OpenWebSpider# v0.1.2

By: SHL

SHL — Sat, 01 Nov 2008 19:47:18 +0000

[cite=shen139]
OWS is structured that it uses the host_id (the ID of the current domain) for many tasks, and so using a regex or any other trick we will destroy the fundamentals of its core.
[/cite]

Ok, then I understand why it’s not an easy solution.

[cite=shen139]
I’m planning a new version of OWS with this feature…
[/cite]

Looking forward to that!

Thanks for your replies!

Kind Regards
// Samuel

By: bernd

bernd — Wed, 29 Oct 2008 13:16:20 +0000

many thanks. I have send you a email.

By: Shen139

Shen139 — Wed, 29 Oct 2008 10:11:29 +0000

Hi bernd,
if a domain is with status = 1 means that that domain has been indexed.
OWS insert new domains with status = 0; status = 1 -> indexed; status = 2 -> indexing!

If you have 0 indexed_pages per domain means that OWS has not been able to index that sites. I could say because there was an error somewhere.
Maybe OWS had an error indexing the first domain and then it wasn’t able to index the rest of the domains.

I don’t speak German and I don’t know what “Der Wartezustand wurde aufgrund eines abgebrochenen Mutex beendet?” is! That error message isn’t inside OWS itselft but in your Framework.

I can read Mutex… so I guess that you had a problem with mutexes.

I can only suggest you to wait for next version (many bug fixes in that version) or email me and I’ll send you a copy.

Stefano

By: bernd

bernd — Wed, 29 Oct 2008 05:23:41 +0000

What means “Der Wartezustand wurde aufgrund eines abgebrochenen Mutex beendet?”

On any domains I have this error.

By: bernd

bernd — Tue, 28 Oct 2008 18:30:27 +0000

Perhaps is my English to bad. I dont know.

“If all the domains in hostlist are indexed OWS won’t have anything to do.”

In my hostlist are 15 domains with status 1 but 0 indexed_pages. When will the spider crawl this? This domains found on a crawl, but the spider has only insert the domain but has no indexed.

“If example.de contains only: example1.de and example2.de OWS will only index these websites, no others! OK?”

Ok, but what is if the example1.de contains the example1A.de ?
example.de
example1.de
example1A.de

Do the spider crawl crawl example.de , example1.de and example1A.de ?

Many thanks

By: Shen139

Shen139 — Tue, 28 Oct 2008 13:41:03 +0000

Using −−add−external OWS will add all hosts different to the current one to the table hostlist!
If example.de contains only: example1.de and example2.de OWS will only index these websites, no others! OK?

What uncrawled domain? Who tells to OWS what to crawl? (the table hostlist)
If all the domains in hostlist are indexed OWS won’t have anything to do.
You should set the status of the domains to: 0
or run OWS to that domain with: openwebspider –index example.de and then: openwebspider –index example1.de and openwebspider –index example2.de

Do you understand?
(Maybe I misunderstood)

By: bernd

bernd — Tue, 28 Oct 2008 13:28:54 +0000

Thanks for reply. I have one more question. The spider find urls on a host and insert this in the database. But not every domain would be crawl by spider. Why ?

I used –add-external .

Example:
spider –> example.de (find external example1.de and external2.de) all internal links are spidered.

in database in table hostlist now I find :
23 example.de 80 1 2008-10-28 45
24 example1.de 80 1 2008-10-28 0
25 example2.de 80 1 2008-10-28 0

The spider says after crawl, that no more domians (or hosts?) are find etc.
bye bye

Could you include a function, that the spider crawl a uncrawled domain from database if he not found a link on the current spidered site?

Sorry for bad english.

By: Shen139

Shen139 — Tue, 28 Oct 2008 09:38:08 +0000

Hi,
Thanks (for the cool spider)!

This is a known bug fixed by OpenWebSpider v0.1.3

I’ll release it as soon as possible (I hope within this week). It fixes many bugs.

I’ve tested your command line arguments with new version and it works fine!

Stefano

By: bernd

bernd — Mon, 27 Oct 2008 17:38:53 +0000

Hello,

this is a cool Spider

I´ve tested this but there is a Errormessage: Getting Urls…Error: Der Index war außerhalb des Arraybereiches.

I tested with this command line.
–index one3p.de -t 10 -r 1 –crawl-delay 5 –req-timeout 30 -l 3000 -s

Another url is working fine but this one works not (http://one3p.de) Whats wrong?

many thanks

By: Shen139

Shen139 — Wed, 22 Oct 2008 14:35:02 +0000

Hi,
you are right, a regex will solve the problem but OWS is structured that it uses the host_id (the ID of the current domain) for many tasks, and so using a regex or any other trick we will destroy the fundamentals of its core.

http://www.domain.com it’s different from domain.com! Many webservers could have different pages for domain with and without WWW. Do you understand? So it’s not simple to solve your way!

I’m planning a new version of OWS with this feature… OWS 0.7 had a command-line argument:
–[
-F

(Free Indexing mode)

1.
This is a new feature in OpenWebSpider v0.6! Free indexing mode means that
the web spider will index pages while it encounter them!
For example if we have a web site with the following structure:

home_page
/ | \
link1 link2 http://www.example.com/linkX

the web spider will index (in order): home_page, link1, link2 and http://www.example.com/linkX
Whereas without this argument the web spider will index only: home_page, link1 and link2
]–
but it wasn’t safe to use and created many problems!