The Main Manual Page Dynamic API Documentation CD-ROM API Documentation About Onix Types About Onix Errors Onix's Web Site at Lextek International Lextek International Onix Full Text Indexing and Retrieval Toolkit

ABOUT DISTRIBUTED INDEXING

Often for large text indexes, such as those produced for the large indexes used to search the web, it is practical to have several machines working on the same index. Onix allows for distributed indexing. Using distributed indexing, you can have several machines, processes, or threads all working on building different parts of the overall index. By doing this, you can increase both the speed of indexing and depending on your needs the flexibility of generating the index itself.

On the other hand, keep in mind that Onix is capable of generating its indexes extremely rapidly -- often far surpassing the speed of the network connection where the data may be originating from. So before deciding to use distributed indexing, run some speed tests on the machine you are planning to use for indexing (or a comparable machine) to see if Onix's indexing speed isn't already fast enough for your application.

A sample call sequence for performing distributed indexing is as follows:

ixStartDistributedIndexingSession()
// Set the file name of the file 
// where the new index data is going to be stored.
ixSetFinalIndexDataPosition()
// Now we commence with the indexing.
while(Not Done)
for(Each Word In Document) {
 ixIndexWord()
}
if(More Data To Index) 
   ixIncrementRecord()
}
ixEndDistributedIndexingSession()
// Now copy the index to wherever it is going
// to be used if it isn't already there.
ixAppendDistributedDataToIndex()

 

SEE ALSO

ixStartDistributedIndexingSession, ixEndDistributedIndexingSession, ixAppendDistributedDataToIndex, ixStartIndexingSession