Skip Ribbon Commands
Skip to main content
Navigate Up
Sign In

Quick Launch

Average Rating:

facebook Twitter
Email
Print Bookmark Alert me when this article is updated

Feedback

ERROR: Relate360 job run_genclusters fails with error as Container [pid=8853,containerID=container_e119_1537088483874_2902_02_000001] is running beyond physical memory limits
Problem Description
The ./run_genclusters.sh job in Relate360 fails, during the CusterGen Group Phase, with error as:

INFO mapreduce.Job: Job job_1537088483874_2902 failed with state FAILED due to: Application application_1537088483874_2902 failed 2 times due to AM Container for appattempt_1537088483874_2902_000002 exited with  exitCode: -104
For more detailed output, check application tracking page:http://localhost:8088/proxy/application_1537088483874_2902/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=8853,containerID=container_e119_1537088483874_2902_02_000001] is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 3.7 GB of 8 GB virtual memory used. Killing container.
Cause
The problem occurs because of the Relate360 job running beyond physical memory limits of the YARN container.
Solution
To change YARN memory for Relate360 jobs, use the -D option. 
For example:
/usr/local/mdmbdrm-10.0/run_genclusters.sh -D yarn.app.mapreduce.am.resource.mb=800 -D yarn.app.mapreduce.am.command-opts=-Xmx1650m --config=config.xml --rule=temdir=/user/cloudera/Output --input=/user/cloudera/null --reducer=1

You can use this -D option for any Relate360 job and customize the YARN memory usage for that job.
More Information
​Dump of the process-tree for container_e119_1537088483874_2902_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 8853 8851 8853 8853 (bash) 0 0 108670976 308 /bin/bash -c /usr/java/latest/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1537088483874_2902/container_e119_1537088483874_2902_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog  -Djava.net.preferIPv4Stack=true -Xmx1717986918 org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/var/log/hadoop-yarn/container/application_1537088483874_2902/container_e119_1537088483874_2902_02_000001/stdout 2>/var/log/hadoop-yarn/container/application_1537088483874_2902/container_e119_1537088483874_2902_02_000001/stderr  
|- 8869 8853 8853 8853 (java) 97760 5324 3866984448 524343 /usr/java/latest/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1537088483874_2902/container_e119_1537088483874_2902_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Djava.net.preferIPv4Stack=true -Xmx1717986918 org.apache.hadoop.mapreduce.v2.app.MRAppMaster 


Exception in thread "main" com.informatica.mdmbde.exception.MDMEnablerException: batch-cluster: Cluster Generation Failed with exception:batch-cluster: MDM Enabler Failed for Job:'ClusterGen Group Phase'
at com.informatica.mdmbde.driver.MDMEnabler.main(MDMEnabler.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: com.informatica.mdmbde.exception.MDMEnablerException: batch-cluster: MDM Enabler Failed for Job:'ClusterGen Group Phase'
at com.informatica.bdrm.setup.handler.GenClusterToolRunner.executeJob(GenClusterToolRunner.java:126)
at com.informatica.bdrm.setup.handler.GenClusterToolRunner.run(GenClusterToolRunner.java:51)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.informatica.mdmbde.driver.MDMEnabler.main(MDMEnabler.java:68)
... 6 more​
Applies To
Product: Relate 360
Problem Type:
User Type:
Project Phase:
Product Version:
Database:
Operating System:
Other Software:

Reference
Attachments
Last Modified Date:11/30/2018 10:21 AMID:561477
People who viewed this also viewed

Feedback

Did this KB document help you?



What can we do to improve this information (2000 or fewer characters)