ixRobotsPermissionGranted -- Returns whether a directory or web page is indexable or not.
BooleanT ixRobotsPermissionGranted(RobotsTxtParserT Parser, char *URL)
Parser -- A robots.txt parser with a robots.txt file loaded.
URL -- The directory and file specified in the URL.
BooleanTrue if the file is traversable and BooleanFalse otherwise.
ixRobotsPermissionGranted will tell you whether you have permission from the web site administrator to download a web page or whether a directory is traversable. ixRobotsPermissionGranted expects to receive the directory and filename in the URL and not the host name, http://, etc. For example, it expects as the URL argument something along the lines of "/netresources/files/index.html" and not "http://www.my.company.com/netresources/files/index.html". (Use ixRobotsPermissionGrantedFullURL() to get permission for the full URL including the host name). ixRobotsPermissionGranted() returns BooleanTrue if you have permission to download a web page and BooleanFalse otherwise.
ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixParseRobotsTxt, ixOutputCompactRobotsTxt, ixRobotsPermissionGrantedFullURL, ixRobotsTxtLength