OpenWebSpider v0.5 Handbook
What is OpenWebSpider?
What license does OpenWebSpider use?
Where can I download OpenWebSpider?
What platforms are supported ?
What I need to compile and run OpenWebSpider?
How do I compile OpenWebSpider (under linux)?
How do I compile OpenWebSpider (under windows with Microsoft Visual C++)?
How do I setup mysql?
What's new in OpenWebSpider v0.5?
Can I convert my existent DB from version 0.4 to 0.5 of OpenWebSpider?
How do I configure OpenWebSpider?
Host rank, page level, page rank?
How can I index my personal Web Site?
What is mod_regexfilter?
Can I write a custom filter module?
And now? How do I set up my Search Service for my web site?
A lot of hosting provider doesn't let you to directly connet to the mysql server! What can I do?
What is OpenWebSpider?
OpenWebSpider is a Web Spider (also known as crawler or web robot) and a search engine!
It can be used to build a search service for a personal web site for internal or related content!
What license does OpenWebSpider use?
OpenWebSpider uses the GNU General Public License (GPL) license and all free software (gcc, mysql, apache and php)
Where can I download OpenWebSpider?
You can download it from http://sourceforge.net/projects/openwebspider/
What platforms are supported ?
Right now the platforms where OpenWebSpider is tested are: Windows and Linux! Maybe you can compile it under other platforms but aren't officially supported
What I need to compile and run OpenWebSpider?
You need gcc or Microsoft Visual C++ to compile it, a mysql server to store data, an apache web server with php support to run the search engine
How do I compile OpenWebSpider (under linux)?
You need mysql library and header (mysql-devel) file!
To compile OpenWebSpider you must type:
$ make
^ this will compile OpenWebSpider, OpenWebSpider Macro and mod_regexfilter
$ make openwebspider
^ will compile only OpenWebSpider
$ make ows_macro
^ will make only OpenWebSpider Macro
$ make mod_regexfilter
^ will make only mod_regexfilter
$ make clean
^ will delete all binary and intermediate files
however you could compile OpenWebSpider manually with:
$ gcc openwebspider-0.5.c -o openwebspider `mysql_config --cflags --libs` -lpthread
OpenWebSpider Macro:
$ gcc macro/ows_macro.c -o ows_macro `mysql_config --cflags --libs`
How do I compile OpenWebSpider (under windows with Microsoft Visual C++)?
Open MS Visual C++; File[Menu] -> Open Workspace... -> Open "openwebspider.dsp";
then you must click: Build[Menu] -> Build openwebspider.exe (OR press F7)
(You need an installation of mysql with header file and you must modify the #include at line 54:
#include "E:\\Programmi\\MySQL\\MySQL Server 5.0\\include\\mysql.h"
with your full path of mysql.h)
How do I setup mysql?
You must install mysql first, then you must set a password for the user root with (Under linux):
# mysqladmin -u root password 'new-password'
Now you can edit mysql configuration file! Intresting fields are:
#The minimum length of the word to be included in a FULLTEXT index
ft_min_word_len=3
#The maximum length of the word to be included in a FULLTEXT index
ft_max_word_len=20
#Use stopwords from this file instead of built-in list
ft_stopword_file= [path] stop-words.txt
# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
key_buffer_size=200M
# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=100M
read_rnd_buffer_size=100M
# This buffer is allocated when MySQL needs to rebuild the index in
# REPAIR, OPTIMZE, ALTER table statements as well as in LOAD DATA INFILE
# into an empty table. It is allocated per thread so be careful with
# large settings.
sort_buffer_size=300M
Now you must compile and execute ows_macro with argument '-c' (usage: -c: Creates all tables needed by OpenWebSpider):
$ ./ows_macro -s localhost -u root -p <password> -c
Server : localhost
Username : root
Db1 : hosts
Db2 : spiderdb
Db3 : temptables
Connecting to mysql...OK
+ Creating database 'hosts'...OK
- Creating table : 'hostlist'...OK
- Creating table : 'rels'...OK
+ Creating database 'spiderdb'...OK
- Creating table : 'pagelist'...OK
+ Creating database 'temptables'...OK
- Creating table : 'tablelist'...OK
What's new in OpenWebSpider v0.5?
OpenWebSpider v0.5 has a lot of codes improvements and intresting features (from changelog):
- Support for external modules (.dll OR .so) argument '-f'
- New argument: '-l' (Limit the maximum number of pages indexed per site) [Default: 0 (No limit)]
- New argument: '-x' (store full html of the indexed page in the DB)
- New argument: '-r' (Saves all relationships between page; who links who and who is linked from)
- New argument: '-u' (index only new pages (Update))
Can I convert my existent DB from version 0.4 to 0.5 of OpenWebSpider?
Yes, You can use OpenWebSpider Macro with the '-a' option:
$ ./ows_macro -s localhost -u root -p <password> -a
Server : localhost
Username : root
Db1 : hosts
Db2 : spiderdb
Db3 : temptables
Connecting to mysql...OK
Converting OpenWebSpider DB from version 0.4 to 0.5...OK
How do I configure OpenWebSpider?
You can configure OpenWebSpider my editing openwebspider.conf:
--------openwebspider.conf--------[
#server1 is the server with the DB 'DB1'
#db1 is the database of the hosts
db1=hosts
mysqlserver1=<server (hostname/IP)>
userdb1=<user>
passdb1=<pass>
#server2 is the server with the DB 'DB2'
#db2 is the database of the indexed pages
db2=spiderdb
mysqlserver2=<server (hostname/IP)>
userdb2=<user>
passdb2=<pass>
#server3 is the server with the DB 'DB3'
#db3 is the database of the temporany tables (used to speed-up the indexing phase)
db3=temptables
mysqlserver3=<server (hostname/IP)>
userdb3=<user>
passdb3=<pass>
#EOF#
]--------openwebspider.conf--------
Host rank, page level, page rank?
The host rank is the sum of the web-sites that link to the considered web-site! For example: if b.com and c.com link to a.com; a.com will have host rank=2
(a.com)
.
./ \.
(b.com) (c.com)
Ex.
A -> B (A links B) (B is linked from A)
B -> A (B links A)
C -> A
Reference Table:
|HostID|LinkedHostID|...(More info)...|
| A | B |...
| B | A |...
| C | A |...
Host : test
HostRank : 5 (example)
MaxPRLev : 10 (definition)
Structure :
p0 Level : 1 PR(p0) : 15
/ | \
p1 p2 p3 : 2 PR(p0,p2,p3) : 10
| | \
p4 p5 p6 : 3 PR(p4,p5,p6) : 8.3
PageRank(px)=HostRank+(MaxPRLev/Level)
PR(p0)=5+(10/1) = 15
PR(p1)=5+(10/2) = 10
...
PR(p5)=5+(10/3) = 8.3
How can I index my personal Web Site?
You can index your web site with OpenWebSpider with:
$ ./openwebspider -i <my_web_site> -e -s
OpenWebSpider(v0.5)
Coded by Shen139
shen139(at)eviltime(dot)com
+ Trying to load openwebspider.conf...OK
- Database1: hosts
- Server1: localhost
- Username DB1: root
- Password DB1: *****
- Database2: spiderdb
- Server2: localhost
- Username DB2: root
- Password DB2: *****
- Database3: temptables
- Server3: localhost
- Username DB3: root
- Password DB3: *****
Start Host: <my_web_site>
Start Page: /
Scan Mode: Index
Mode: Single Host
Max depth level 0
Mysql server n.1: localhost
Mysql server n.2: localhost
Mysql server n.3: localhost
Surfing the net... (press CTRL+C to exit)
Connecting to Mysql server n.1 (localhost)...OK
Connecting to Mysql server n.2 (localhost)...OK
Connecting to Mysql server n.3 (localhost)...OK
Deleting old indexed pages from <my_web_site>...OK
Creating thread 20 of 20
(0) Current -> http://<my_web_site>/robots.txt (http://<my_web_site>)
- HTTP header: HTTP/1.1 302 Found
- Location: http://<my_web_site>/not_found.html
- Downloaded 0 Kb (175 bytes) in 1000 ms
- Nothing to do with robots.txt
(0) Current -> http://<my_web_site>/ (http://<my_web_site>)
- HTTP header: HTTP/1.1 200 OK
- Downloaded 4 Kb (4694 bytes) in 0 ms
- Checked in 0 ms (11 URL found)
- Indexed in 0 ms
(1) Current -> http://<my_web_site>/not_found.html ()
- HTTP header: HTTP/1.1 200 OK
- Downloaded 1 Kb (1681 bytes) in 0 ms
- Checked in 0 ms (3 URL found)
- Indexed in 0 ms
(1) Current -> http://<my_web_site>/page1.html (page n.1)
- HTTP header: HTTP/1.1 200 OK
- Downloaded 3 Kb (3072 bytes) in 0 ms
- Checked in 0 ms (3 URL found)
- Indexed in 0 ms
(0) Current -> http://<my_web_site>/page2.html (Page n.2)
- HTTP header: HTTP/1.1 200 OK
- Downloaded 4 Kb (4919 bytes) in 0 ms
- Checked in 0 ms (18 URL found)
- Indexed in 0 ms
(3) Current -> http://<my_web_site>/map.html (Map)
- HTTP header: HTTP/1.1 200 OK
- Downloaded 2 Kb (2083 bytes) in 0 ms
- Checked in 0 ms (3 URL found)
- Indexed in 0 ms
(6) Current -> http://<my_web_site>/docs.txt (document)
- HTTP header: HTTP/1.1 200 OK
- Downloaded 1 Kb (1965 bytes) in 0 ms
- Indexed in 0 ms
(7) Current -> http://<my_web_site>/sourcecode.c (test)
- HTTP header: HTTP/1.1 200 OK
- Downloaded 2 Kb (2646 bytes) in 0 ms
- Indexed in 1000 ms
::::::::::::::::::
Killing Threads...
Threads killed
Swapping temporany table to the index...OK
Flushing temporany data...OK
+ HostRank: 0
- Calculating PageRank...OK
+ STATS
- Host: <my_web_site>
- Pages: 21
- Downloaded: 83 Kb
- Scan time: 6s (14:52:1 - 14:52:7)
Deleting temporany data...OK
Dropping temporany data...OK
Bye
The '-i' argument followed by an URL tells to OpenWebSpider to index that site!
'-s' is the single host mode, OpenWebSpider will index only that site!
'-e' tells to do not add external host to the DataBase! External URLs are considered all that URLs with different hostname from the Start-URL
What is mod_regexfilter?
mod_regexfilter is a loadable module that let you filter the pages you want to index by a regular expression!
Examples:
1) you want index only html pages (.htm OR .html)! You must edit mod_regex.conf as follows:
--------mod_regex.conf--------[
page=\.(htm|html)$
]--------mod_regex.conf--------
2) you want index only .txt, .c, .cpp and .h files form italian domains (eg. www.host.it/test.c)
--------mod_regex.conf--------[
hostname=\.(it)$
page=\.(txt|c|cpp|h)$
]--------mod_regex.conf--------
3) you want index only pages beginning for "/jobs.php" (eg. www.host.com/jobs.php?id=139)
--------mod_regex.conf--------[
hostname=
page=^(/jobs\.php)
html=
text=
]--------mod_regex.conf--------
As you can see in these examples you can filter the hostname, the page, the (pure) html and the text of a page! Then you can execute OpenWebSpider
(Under Linux) ($ ./openwebspider -i www.openwebspider.org -f modules/regexFilter.so):
OpenWebSpider(v0.5)
Coded by Shen139
shen139(at)eviltime(dot)com
+ Trying to load openwebspider.conf...OK
- Database1: hosts
- Server1: 192.168.0.1
- Username DB1: root
- Password DB1: *****
- Database2: spiderdb
- Server2: 192.168.0.1
- Username DB2: root
- Password DB2: *****
- Database3: temptables
- Server3: 192.168.0.1
- Username DB3: root
- Password DB3: *****
Trying to open module: modules/regexFilter.so...OK
+ Trying to import functions:
- modFilter() ...loaded
- modFilter() has init function modInitFilter():...yes
- modInitFilter()...OK
- modHandleExtension() ...not found
....
(Under windows) (Command line: -i http://www.openwebspider.org -f regexFilter.dll):
OpenWebSpider(v0.5)
Coded by Shen139
shen139(at)eviltime(dot)com
+ Trying to load openwebspider.conf...OK
- Database1: se_hosts
- Server1: localhost
- Username DB1: root
- Password DB1: *****
- Database2: se_spiderdb
- Server2: localhost
- Username DB2: root
- Password DB2: *****
- Database3: se_tmp
- Server3: localhost
- Username DB3: root
- Password DB3: *****
Trying to open module: regexFilter.dll...OK
+ Trying to import functions:
- modFilter() ...loaded
- modFilter() has init function modInitFilter():...yes
- modInitFilter()...OK
- modHandleExtension() ...not found
...
Can I write a custom filter module?
Yes, You can! To write a module for OpenWebSpider is quite easy (and I hope to make it easier in the future)! Here is a skeleton for a filter module loadable by OpenWebSpider:
1) You should include modHeader.h! this is an header file containing the structure needed to share data from your module and OpenWebSpider
2) You may want have an initializer for your module! For example a routine that check if a file exists or parse a configuration file and initialize some variables! You must call this function: modInitFilter
int modInitFilter (char* hostname, char* error)
{
/*on OK: return 1*/
return 1;
/* on ERROR: return 0 - you can use the variabile error to report what happened
strcpy(error,"test");
return 0;
*/
}
3) You must write the main of your filter module! For example you want a filter that checks if the text of the current page contains the word "shen139":
/* modFilter should return 1 if the current page must be indexed 0 if discarded*/
int modFilter (struct functArg* arg)
{
/*
this example function checks if the text of the current page has the string "shen139" if yes tells to the webspider to don't index the page otherwise index the page
*/
if(arg)
if(strstr(arg->text,"shen139")>arg->text)
return 1;
return 0;
}
"struct functArg* arg" ?!?
struct functArg* arg is a structure passed to the filter module by openwebspider! it's filled with all those information needed to build a useful filter (you can find it in modules/modHeader.h):
- hostname (arg->hostInfo->Host)
- current page (arg->hostInfo->Page)
- Page title (<title>Description</title>) (arg->hostInfo->Description)
- web server port (arg->hostInfo->port)
- page type: 1=HTML file(.htm,.html,.php,.asp,...); 2=text file(.txt,.c,.h,...); 3=unhandled extensions (.exe,.doc,.swf,...); (arg->hostInfo->type)
- page level (arg->hostInfo->level)
- HTML code of the page (arg->html)
- text of the page (arg->text)
- the number of the pages indexed (arg->hostInfo->PagesViewed)
- the amount of downloaded bytes (arg->bytesDownloaded)
In addition this structure provides 3 pointer to the mysql connection enstablished with the 3 databases:
void* mysqlDB1;
void* mysqlDB2;
void* mysqlDB3;
Under linux you can compile a custom filter module with:
gcc -g -c myModFilter.c
gcc -g -shared -W1,-soname,myModFilter.so.0 -o myModFilter.so myModFilter.o -lc
Under windows with MS visual c++ you must open a new Project Win32 Dinamic-link Libray and Build it from the menu: Build or by pressing F7
And now? How do I set up my Search Service for my web site?
First you need a web server(suggested apache) with PHP and mysql support!
You can also use MS IIS and ASP (I tested it in the past) but isn't officially supported!
After you have configured apache and mysql and PHP you should write a page that query the database!
I wrote one that executes simply full-text queries
-----------vars.php-------------[
<?php
/*MISC*/
$ResultsPerPage=10;
$Logging=1;
/*
* 0= don't log
* 1= log query.log
* 2= log mysql (You must create table querylist in the spiderdb database)
CREATE TABLE `querylist` (
`idquery` int(11) NOT NULL auto_increment,
`query` varchar(255) NOT NULL,
`results` int(5) NOT NULL,
`date` varchar(10) NOT NULL,
`time` varchar(10) NOT NULL,
`ipaddr` varchar(20) NOT NULL,
`user` varchar(200) NOT NULL,
PRIMARY KEY (`idquery`)
) ENGINE=MyISAM;
*/
/*Connection*/
$server="mysql.server";
$user="username";
$pass="password";
$database="database";
?>
]-----------vars.php-------------
-----------searchfunct.php-------------[
<?php
function owssize($owsQuery)
{
$tmpowsQuery=" " . trim($owsQuery) . " " ;
if(strstr($tmpowsQuery," \"")==FALSE && strstr($tmpowsQuery," +")==FALSE && strstr($tmpowsQuery," -")==FALSE && strstr($tmpowsQuery,"<")==FALSE && strstr($tmpowsQuery,">")==FALSE && strstr($tmpowsQuery,"* ")==FALSE)
{
if(strstr($owsQuery," ")==FALSE)
{
$tmpowsQuery="\\\"".trim($owsQuery)."\\\"";
$owsQuery=$tmpowsQuery;
}
else
{
$tmpowsQuery="+\\\"".trim($owsQuery);
$owsQuery=str_replace(" ","\\\" +\\\"",$tmpowsQuery);
$owsQuery.="\\\"";
}
}
return $owsQuery;
}
function DrawPages($page,$results,$resultsperpage,$query)
{
$lbound=0;
$rbound=0;
if($page<0)
$page=0;
if($page-10<0) $lbound=0; else $lbound=$page-10;
if($page+10>$results/$resultsperpage)
$rbound=$results/$resultsperpage;
else
$rbound=$page+10;
for($i=$lbound;$i<$rbound;$i++)
{
if($i!=$page)
printf("<font face=\"Arial\"><a href=\"search.php?search=%s&page=%d\">%d</a></font>\r\n",urlencode($query),$i,$i+1);
else
printf("<font face=\"Arial\"><b>%d </b></font>",$i+1);
}
}
function microtime_float()
{
list($usec, $sec) = explode(" ", microtime());
return ((float)$usec + (float)$sec);
}
function fSearch($search,$page,$server,$user,$pass,$database,$log,$ResultsPerPage)
{
$my_search=mysql_escape_string($search);
$db = mysql_connect($server, $user, $pass);
if ($db == FALSE)
die ("Error 1 [mysql_connect()]: Can't connect to mysql server");
mysql_select_db($database, $db)
or die ("Error 2 [mysql_select_db()]");
if($page=="")
$page=0;
$start = microtime_float();
$query = "select max(match(html,description,hostname,page) against(\"".owssize($my_search)."\" in boolean mode)) from pagelist where match(html,description,hostname,page) against(\"".owssize($my_search)."\" in boolean mode)" ;
$result = mysql_query($query, $db);
$row = mysql_fetch_array($result);
$minword=$row[0];
if($minword==0)
$minword=1;
$query = "select count(*) from pagelist where match(html,description,hostname,page) against(\"".owssize($my_search)."\" in boolean mode) >=".(int)abs($minword);
$result = mysql_query($query, $db);
$row = mysql_fetch_array($result);
$resultsfound=$row[0];
if( $page>$resultsfound/$ResultsPerPage)
$page=(int)abs($resultsfound/$ResultsPerPage);
$query = "select hostname,page,description,match(html,description,hostname,page) against(\"".owssize($my_search)."\") + match(description) against(\"".owssize($my_search)."\") as relevancy,match(html,description,hostname,page) against(\"".owssize($my_search)."\" in boolean mode) as wrdcount,idpage,rank, LEFT(html, 400),level from pagelist where match(html,description,hostname,page) against(\"".owssize($my_search)."\" in boolean mode) >=".(int)abs($minword)." order by wrdcount DESC,relevancy DESC,rank DESC limit ".($page*$ResultsPerPage).",".$ResultsPerPage ;
$result = mysql_query($query, $db);
$nRes= mysql_num_rows($result);
$endtime = abs(number_format(microtime_float()-$start,3));
?>
<div align="center">
<table border="0" width="100%" cellspacing="0" cellpadding="0">
<tr>
<td style="border: 0px solid ; font-size: 10px; font-family: Verdana;" width="100%" height="3" bgcolor="#99FFCC"> Query:
<?=htmlspecialchars($search)?>
-
<?=$resultsfound?>
results found in
<?= $endtime ?>
seconds </td>
</tr>
</table>
</div>
<?
while ($row = mysql_fetch_array($result))
{
$mText=strtolower(htmlspecialchars($row[7]));
$count=str_word_count($search);
$words=str_word_count(strtolower($search),1);
for($i=0;$i<$count;$i++)
{
$new_text=str_replace($words[$i],"<strong>".$words[$i]."</strong>", $mText);
$mText=$new_text;
}
?>
<table border="0" width="100%" cellspacing="0" cellpadding="0">
<tr>
<td style="border: 0px solid ; font-size: 10px; font-family: Verdana;" width="100%" height="3" bgcolor="#EAFEFF"> <B><a href="http://<? echo"$row[0]$row[1]" ?>">
<?= htmlspecialchars($row[2])?>
</a></B> <br>
<?= $mText ?>
... <br>
<font color="#006600">Relevancy:
<?= number_format($row[3],2)?>
- Word found: <? echo"$row[4]"?> - Level: <? echo"$row[8]" ?> - Rank: <? echo"$row[6]" ?> <font face="Arial" size="1"><a href="cache_txt.php?ID=<? echo"$row[5]" ?>">Text cache</a></font></font>
</p>
</tr>
</table>
<br>
<?
}
echo "<p align=\"center\">";
DrawPages($page,$resultsfound,$ResultsPerPage,htmlspecialchars($search));
echo "</p>";
$ip=getenv("REMOTE_ADDR");
if($log==1)
{
$var=fopen("query.log","a");
if($var!=NULL)
{
$tm=strftime ("%A %d %B %Y %H:%M");
$log= "$tm\t\t$ip\t\t$search\r\n";
fwrite($var,$log);
fclose($var);
}
}
else
if($log=2)
{
$query = "INSERT INTO querylist (query,results,date,time,ipaddr) VALUES('$my_search',$resultsfound,curdate(),curtime(),'$ip')";
$result = mysql_query($query, $db);
}
mysql_close($db);
}
?>
]-----------searchfunct.php-------------
-----------search.php-------------[
<html>
<head>
<title>OpenWebSpider - PHP Example</title>
</head>
<body vlink="#0000FF" alink="#0000FF">
<? include 'vars.php'; ?>
<? include 'searchfunct.php'; ?>
<table width="500" border="0" align="center" cellpadding="0" cellspacing="0" bgcolor="#66FFCC">
<tr>
<td> <p align="center"> <font face="Arial"><a href="search.php">Search</a></font></p>
<form action="search.php" method="get">
<div align="center"><font face="Arial">
<input maxLength="100" size="60" value="<?= htmlspecialchars(stripslashes($HTTP_GET_VARS["search"])); ?>" name="search">
<input type="submit" value="Search" >
</p>
</font> </div>
</form></td>
</tr>
</table>
<div align="left">
<?
$search=stripslashes($HTTP_GET_VARS["search"]);
$page=$HTTP_GET_VARS["page"];
if($search!="")
fSearch($search,$page,$server,$user,$pass,$database,$Logging,$ResultsPerPage);
?>
</div>
</body>
</html>
]-----------search.php-------------
-----------cache.php-------------[
<? include 'vars.php'; ?>
<?
$id=$HTTP_GET_VARS["ID"];
if($id=="")
die("Error 1");
$db = mysql_connect($server, $user, $pass);
if ($db == FALSE)
die ("Error 2");
mysql_select_db($database, $db)
or die ("Error 3");
$query = "select hostname,page,date,time,html,htmlcache from pagelist where idpage=".$id;
$result = mysql_query($query, $db);
if($result==0)
die("Error 4");
$row = mysql_fetch_array($result);
echo "<meta http-equiv=\"Content-Type\" content=\"text/html; charset=ISO-8859-1\">\r\n<BASE HREF=\"http://".$row[0]."/\">\r\n"
?>
<div align="center" style="background-color:#00FFCC;"><span style="font-family: Verdana, Arial, Helvetica, sans-serif;
font-size: 18px; ">OpenWebSpider Cache</span><br>
<center>
<strong>http://
<?= $row[0].$row[1] ?>
</strong> - Text cache of
<?= $row[2]." - ".$row[3] ?>
<br>
<a href="javascript:history.back();">Back</a> - <a href="http://www.openwebspider.org/">OpenWebSpider Home</a> - <a href="http://www.openwebspider.org/feedback.php">Feedback</a> - <a href="http://www.openwebspider.org/openwebspider_download.php">OpenWebSpider Download</a>
<hr>
</div>
<br>
<?
if($row[5]=='')
echo "<P align=left>".$row[4]."</P>";
else
echo "\r\n\r\n".$row[5];
mysql_close($db);
?>
]-----------cache.php-------------
A lot of hosting provider doesn't let you to directly connet to the mysql server! What can I do?
OpenWebSpider needs to be directly connected to a mysql server and if your hosting provider let you connect only via phpMyAdmin or similar you should use an annoying trick! You should setup a mysql server in your computer and you must index there your web-site, then you should dump the spiderdb database in a .sql file and then you should import the dump via phpMyAdmin
