ixParseRobotsTxt -- Use the robots.txt parser to parse a web site's robots.txt file
void ixParseRobotsTxt(RobotsTxtParserT Parser, char *RobotsTxt);
Parser -- A robots.txt parser which was created by a call to ixCreateRobotsTxtParser().
RobotsTxt -- A pointer to a buffer containing a robots.txt file. The robots.txt must be NULL terminated.
Nothing.
ixParseRobotsTxt tells the parser to parse the NULL terminated robots.txt contained in the buffer RobotsTxt. It is important that you set your robot's name with a call to ixSetRobotName() before you parse the robots.txt file so the parser will know which instructions pertain to your crawler. Note that a robots.txt parser is good for only 1 robots.txt file. Do not try to use a parser on multiple robots.txt files. (It is better to create a new one and dispose of the old ones as you do your web crawling.)
ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixOutputCompactRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL, ixRobotsTxtLength