Skip Ribbon Commands
Skip to main content
Navigate Up
Sign In

Quick Launch

Average Rating:

facebook Twitter
Email
Print Bookmark Alert me when this article is updated

Feedback

Running the consolidate job in incremental mode corrupts the existing cluster in Relate 360
Problem Description
In Relate 360, the data is loaded to HDFS in 2 iterations. The first iteration was initial load and the second iteration was incremental load.

Initial Data Load :

/usr/local/mdmbdrm-10.0/run_genclusters.sh --config=./Config.xml --rule=./IndividualMatchRule.xml --input=/user/infa/Attachments/employee2.txt --reducer=3 --hdfsdir=/user/infa/Attachments/run2 --outputpath=/user/infa/Attachments/run2/output2 

/usr/local/mdmbdrm-10.0/run_clusterload.sh --config=./Config.xml --rule=./IndividualMatchRule.xml --input=/user/infa/Attachments/run2/output2/batch-cluster/output/dir/pass-join --reducer=3 --hdfsdir=/user/infa/Attachments/run2/

/usr/local/mdmbdrm-10.0/run_consolidate.sh --config=./Config.xml --consolidate=./MDMBDRMConsolidationRule.xml --input=/user/infa/Attachments/run2/output2/batch-cluster/output/dir/pass-join --reducer=3 --hdfsdir=/user/infa/Attachments/run2/ --


After the initial data load, the PR table has 2 records based on sys0 column.

Incremental data Load: One record has been added which is same with sys1 column. So after running this there should be only 2 records. However, after incremental data load the number of records becomes 3.


/usr/local/mdmbdrm-10.0/run_genclusters.sh --config=./Config.xml --rule=./IndividualMatchRule.xml --input=/user/infa/Attachments/employee2.txt --reducer=3 --hdfsdir=/user/infa/Attachments/run2 --outputpath=/user/infa/Attachments/run2/output2 --incremental


/usr/local/mdmbdrm-10.0/run_clusterload.sh --config=./Config.xml --rule=./IndividualMatchRule.xml --input=/user/infa/Attachments/run2/output2/batch-cluster/output/dir/pass-join --reducer=3 --hdfsdir=/user/infa/Attachments/run2/ --incremetal

/usr/local/mdmbdrm-10.0/run_consolidate.sh --config=./Config.xml --consolidate=./MDMBDRMConsolidationRule.xml --input=/user/infa/Attachments/run2/output2/batch-cluster/output/dir/pass-join --reducer=3 --hdfsdir=/user/infa/Attachments/run2/ --incremetal.



Cause
​In the case of Incremental load, the consolidate option is not there in the gencluster command. Also, the consolidate has extra option incremental which should not be there.
Solution
​To achieve this expected function, run the incremental load with the following commands:

/usr/local/mdmbdrm-10.0/run_genclusters.sh --config=./Config.xml --rule=./IndividualMatchRule.xml --input=/user/infa/Attachments/employee2.txt --reducer=3 --hdfsdir=/user/infa/Attachments/run2 --outputpath=/user/infa/Attachments/run2/output2 --incremental --consolidate

There should be extra switch in the gencluster command.

/usr/local/mdmbdrm-10.0/run_clusterload.sh --config=./Config.xml --rule=./IndividualMatchRule.xml --input=/user/infa/Attachments/run2/output2/batch-cluster/output/dir/pass-join --reducer=3 --hdfsdir=/user/infa/Attachments/run2/ --incremetal

/usr/local/mdmbdrm-10.0/run_consolidate.sh --config=./Config.xml --consolidate=./MDMBDRMConsolidationRule.xml --input=/user/infa/Attachments/run2/output2/batch-cluster/output/dir/pass-join --reducer=3 --hdfsdir=/user/infa/Attachments/run2/

There should not be any incremental switch in the consolidate command.
More Information
Applies To
Product: Relate 360
Problem Type: Configuration
User Type: Developer
Project Phase: Configure
Product Version:
Database:
Operating System:
Other Software:

Reference
Attachments
Last Modified Date:12/25/2019 10:37 PMID:590666
People who viewed this also viewed

Feedback

Did this KB document help you?



What can we do to improve this information (2000 or fewer characters)