ixRobotsTxtLength -- The approximate length that an outputted robots.txt will be.
size_t ixRobotsTxtLength(RobotsTxtParserT Parser);
Parser -- A robots.txt parser which has a robots.txt file read into it.
The approximate number of bytes a robots.txt file would be when output.
ixOutputCompactRobotsTxt() allows you to output a "compact" robots.txt which only has instructions which apply to your web crawler and without any of the comments which occurred in the original robots.txt file. Use ixRobotsTxtLength() to find out how large your memory buffer needs to be to hold this compact robots.txt. This function only returns an approximate length and so it is a good idea to add a few bytes to the value returned from ixRobotsTxtLength() before allocating and using any memory for ixOutputCompactRobotsTxt().
ixOutputCompactRobotsTxt, ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixParseRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL