The Main Manual Page Dynamic API Documentation CD-ROM API Documentation About Onix Types About Onix Errors Onix's Web Site at Lextek International Lextek International Onix Full Text Indexing and Retrieval Toolkit

ixParseRobotsTxt

NAME

ixParseRobotsTxt -- Use the robots.txt parser to parse a web site's robots.txt file

SYNOPSIS

void ixParseRobotsTxt(RobotsTxtParserT Parser, char *RobotsTxt);

ARGUMENTS

Parser -- A robots.txt parser which was created by a call to ixCreateRobotsTxtParser().

RobotsTxt -- A pointer to a buffer containing a robots.txt file. The robots.txt must be NULL terminated.

RETURNS

Nothing.

DESCRIPTION

ixParseRobotsTxt tells the parser to parse the NULL terminated robots.txt contained in the buffer RobotsTxt.  It is important that you set your robot's name with a call to ixSetRobotName() before you parse the robots.txt file so the parser will know which instructions pertain to your crawler.  Note that a robots.txt parser is good for only 1 robots.txt file.  Do not try to use a parser on multiple robots.txt files.  (It is better to create a new one and dispose of the old ones as you do your web crawling.)

SEE ALSO

ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixOutputCompactRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL, ixRobotsTxtLength