Their typical
x
indexing process using Verity indexing tools took about 96 hours. The case was similar with
other commercial indexing tools as the SBC staff indicated at that time. The challenge for
Verity was obvious; the 96 hours of indexing time should be knocked down to 10 hours to
win that deal. As a proof of concept engagement, I wrote a multiprocessing indexing tool
that utilized their quad-CPU server. The following steps summarize what I did for them:
??? On paper, I divided their meta-content into 20 blocks, each of about 500 thousand
records.
??? Twenty listeners were started on their Oracle database server, and each listener was
associated with one of the above-mentioned 20 blocks.
??? I wrote a short script (e.g., managing script) to trigger four instances of the indexer
to start working, each on its associated block, and then sleep for a specific time.
o Once an indexer instance completed dumping its block, it indexed that into a
distinct collection (i.e., mini searchable index).
o The managing script would then sleep for a specific time (i.e., 40 minutes) before
it triggered the next four instances of the indexer to start their work. This wait-time
was necessary to avoid congestion on the processing cycles of the four CPUs.
Pages:
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27