Scalable Import issue with HPSD 4.5 - Not all Servants are Closing
I am looking for help with a Scalable Import on HPSD 4.5 SP 18. We are trying to Sequential Import of items, and need to use Parallel processing because of the size of the imports. These imports must be sequential because the second set of data being imported is dependent on the first set.
We have tried doing this two ways.
First we attempted to split the imports into separate commands, running from within one batch script. We spawned several servants, and then started the Master Process. We also made sure to set the -stopservants option in the Master command.
The problem is that if the Master Servent finished processing an XML chunk, and there we no remaining chunks, it would close, and close any Servants that were also finished processing. The Servents still processing would continue to run, and would remain open even when they were finished processing. This prevented our batch script from going to the subsequent data import command.
Secondly, we tried running both imports without closing down the servants until after the second import had finished. This time, when the Master process finished a chunk and had no more chunks left from the first import, it started to process the second import, without waiting for the first import to complete. This is a problem because we need to run these import sequentially.
Ideally we would like to set up our number of XML chunks so that the Master process would always pick up the last chunk, but as the size of our data source will be changing, there's no way we can do this.
Does anyone know of a way to force the Master Process to reamin active until all of the XML chunks for a particular import have completed.?