Comments on: OpenWebSpider# v0.1.3 http://www.openwebspider.org/2008/11/05/openwebspider-v013/ Open Source Web Spider Tue, 19 May 2009 07:59:47 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Shen139 http://www.openwebspider.org/2008/11/05/openwebspider-v013/comment-page-1/#comment-699 Shen139 Thu, 11 Dec 2008 13:16:33 +0000 http://www.openwebspider.org/?p=96#comment-699 Hi, As you can see in openwebspider.con: --[ # used to influence the query that choose the domains to index # SELECT id FROM hostlist WHERE status=0 AND ( <$sql_hostlist_where here$> ) ORDER by priority DESC # sql_hostlist_where is "1=1" by default # Examples: # index only ".com" domains # sql_hostlist_where = hostname LIKE '%.com' # index only domains on port 8080 # sql_hostlist_where = port = 8080 # index only a portion of the database; from ID 600 to 2000 # sql_hostlist_where = ID >= 600 AND ID <= 2000 ]-- This is useful if you run many crawlers and you want that each crawler index only a fixed amount of domains! Stefano Hi,
As you can see in openwebspider.con:
–[
# used to influence the query that choose the domains to index
# SELECT id FROM hostlist WHERE status=0 AND ( < $sql_hostlist_where here$> ) ORDER by priority DESC
# sql_hostlist_where is "1=1" by default
# Examples:
# index only ".com" domains
# sql_hostlist_where = hostname LIKE '%.com'
# index only domains on port 8080
# sql_hostlist_where = port = 8080
# index only a portion of the database; from ID 600 to 2000
# sql_hostlist_where = ID >= 600 AND ID <= 2000
]–

This is useful if you run many crawlers and you want that each crawler index only a fixed amount of domains!
Stefano

]]> By: bernd http://www.openwebspider.org/2008/11/05/openwebspider-v013/comment-page-1/#comment-288 bernd Thu, 06 Nov 2008 12:53:48 +0000 http://www.openwebspider.org/?p=96#comment-288 Hi, how works it? "New feature: New configuration file field: sql_hostlist_where" thanks Hi,

how works it?
“New feature: New configuration file field: sql_hostlist_where”

thanks

]]>