The Main Manual Page Dynamic API Documentation CD-ROM API Documentation About Onix Types About Onix Errors Onix's Web Site at Lextek International Lextek International Onix Full Text Indexing and Retrieval Toolkit

ixOutputCompactRobotsTxt

NAME

ixOutputCompactRobotsTxt -- Output a compact form of robots.txt to a buffer.

SYNOPSIS

void ixOutputCompactRobotsTxt(RobotsTxtParserT Parser, char *RobotsTxt);

ARGUMENTS

Parser -- A robots.txt parser created with ixCreateRobotsTxtParser() which has parsed a robots.txt file.

RobotsTxt -- A pointer to buffer large enough to hold the compact robots.txt

RETURNS

Nothing.

DESCRIPTION

robots.txt allows web administrators to control which robots access their web sites and how the web site is to be crawled.  ixOutputCompactRobotsTxt() outputs a shorted form of robots.txt to a buffer which applies only to your robot.  This shorted form does not contain instructions for other robots or any comments found within the original robots.txt file and is suitable for storing in a database to reduce network usage during the web crawling.  The buffer RobotsTxt should be at least ixRobotsTxtLength() long and preferably a bit longer.  The compact form of robots.txt will be placed in the RobotsTxt buffer and will be 0 (0x00)  terminated.

SEE ALSO

ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixParseRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL, ixRobotsTxtLength