Skip Ribbon Commands
Skip to main content
Navigate Up
Sign In

Quick Launch

Average Rating:

facebook Twitter
Email
Print Bookmark Alert me when this article is updated

Feedback

ERROR: "[[DtmDTF_0001] Data Transport Error, Origin :[IProxyDTMService :: getSessionExecutionState RPC]" while running a mapping in Blaze Execution mode using Informatica BDM
Problem Description

While running a mapping in 'Blaze' execution mode using Informatica BDM, mapping execution fails. Following error messages could be observed in the mapping run log and the 'Blaze Grid Manager' application log: 

 

Informatica Mapping Log Trace

 

2019-01-21 15:10:08.157 <LdtmWorkflowTask-pool-1-thread-7> INFO: [task MAINSESSION_task1]: [GCL_21] The grid task [gtid-1010-1-01403059-1] is submitted to the application ID [application_1549193893398_1025].
2019-01-21 15:10:08.157 <LdtmWorkflowTask-pool-1-thread-7> INFO: [task MAINSESSION_task1]: [GCL_5] Logs for the grid task [gtid-1010-1-01403059-1] appear in the following location on the Hadoop cluster: [/tmp/infa/blazelogs/application_1549193893398_1025/taskletlogs/gtid-1010-1-01403059-1].
2019-01-21 15:10:08.157 <LdtmWorkflowTask-pool-1-thread-7> WARNING: [task MAINSESSION_task1]: [GCL_22] If local log directory for the node on the cluster does not have the correct permissions, the local logs for the Blaze engine will appear in the current working directory of the container.
2019-01-21 15:11:18.811 <DTFPool-15-thread-28> INFO: [task MAINSESSION_task1]: [GCL_19] The grid task [gtid-1010-1-01403059-1] has the tracking URL [
http://inglxyz05.informat​ica.com:9080/Blaze?tasktype=gridtask&id=gtid-1010-1-01403059-1&isParent=false].
2019-01-21 15:11:18.811 <DTFPool-15-thread-28> INFO: [task MAINSESSION_task1]: [GCL_26] High availability is disabled.
2019-01-21 15:12:00.620 <DTFPool-15-thread-27> SEVERE: [task MAINSESSION_task1]: [GCL_4]
The grid task [gtid-1010-1-01403059-1] execution failed.

2019-01-21 15:12:00.620 <DTFPool-15-thread-27> INFO: [task MAINSESSION_task1]: [GCL_6] The grid task [gtid-1010-1-01403059-1] has the following statistics:
    
         Tasklets                             : [153]
              successfully executed tasklets       : [2]
              failed tasklets                      : [4]
              cancelled tasklets before execution  : [0]
              cancelled tasklets during execution  : [147]
2019-01-21 15:12:00.621 <DTFPool-15-thread-27> INFO: [task MAINSESSION_task1]: [GCL_8] The following error appears for all failed tasklets in the grid task [gtid-1010-1-01403059-1]: [
The tasklet [gtid-1010-1-01403059-1_s0_t-131] failed with the following error: [com.informatica.dtm.transport.DTFUncheckedException: [[DtmDTF_0001] Data Transport Error, Origin :[IProxyDTMService :: getSessionExecutionState RPC].]

 

Blaze Grid Manager Application - stderr log

 

2019-01-21 15:10:09.125 <AMRM Callback Handler Thread> INFO: New container response: id [container_e87_1547173892358_1010_01_000019]; capability [<memory:5120, vCores:4>]; node [datanodetst04.allegistest.com]; priority [6]..

2019-01-21 15:10:09.179 <pool-5-thread-5> INFO: Attempting to start native container [DTMProcess_1] with container id [container_e87_1547173892358_1010_01_000019] on host [inglxyz07.informatica.com]..

2019-01-21 15:11:51.046 <AMRM Callback Handler Thread> INFO: Container completion status: id [container_e87_1547173892358_1010_01_000019]; state [COMPLETE]; diagnostics [Container [pid=15398,containerID=container_e87_1547173892358_1010_01_000019] is running beyond physical memory limits. Current usage: 5.0 GB of 5 GB physical memory used; 9.8 GB of 10.5 GB virtual memory used. Killing container.

Dump of the process-tree for container_e87_1547173892358_1010_01_000019 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE​​​

Cause

The encountered issue generally occurs when the mapping processes huge volume of data and available resource configuration of Blaze DTM container is not sufficient to complete the tasks. 


Mapping executed in Blaze Engine would be subdivided into 'Tasklets', when the execution plan is generated successfully after submission. Tasklet would be the smallest unit of the mapping execution. 

 

By default, '5GB' and '4 CPU cores' would be used for the DTM process, created by the 'Blaze Engine' during the mapping execution. With the setting of 'Memory=5120, vcore=4'  for the DTM Process, maximum of '4' Tasklets would be run in the DTM container and the available memory of '5120 MB' would be divided among the 4 Tasklets. So effectively, each Tasklet would have around '1280 MB' (5120/4=1280) of memory, available for execution. If the available '1280 MB' memory is not sufficient for the Tasklet to complete the execution, then the Tasklet would fail and that in turn, would lead to mapping execution failure.


To resolve the issue, it would be required to either increase the memory configuration through 'infagrid.orch.scheduler.oop.container.pref.memory' attribute, or decrease the CPU cores through 'infagrid.orch.scheduler.oop.container.pref.vcore' configuration. Both the actions, increasing container memory and decreasing CPU cores, could also be performed, if required.

Solution

To resolve the issue, perform the following steps:

 

  • Determine the resources available in the Hadoop cluster for the execution of Blaze jobs, after Blaze Engine startup. For more information on the resource requirement to start Blaze Engine, refer to KB 533143.
  • Based on the resources available in the Hadoop cluster for the execution of Blaze jobs, reconfigure the DTM process memory and CPU configuration.

 

Perform the following steps for reconfiguring the memory and CPU configurations of 'DTM processes', created by Blaze Engine, depending on the Informatica BDM version:

 

From Informatica 10.2.1 version:

 

  1. Login to Informatica Administrator console or launch Informatica Developer client.
  2. Navigate to 'Connections' tab in case of Informatica Administrator console and 'Windows > Preferences > Connections > [Domain]> Cluster', when Developer client is used.
  3. Select the 'Hadoop Pushdown' connection being used for running the jobs.
  4. Navigate to 'Blaze Engine' section.
  5. Edit the 'Advanced Properties' attribute in the section.
  6. Update the values of the following attributes under 'Advanced Properties' section, by adding 2-4G to the existing memory configuration (or) decreasing the CPU core configuration:

 

 

infagrid.orch.scheduler.oop.container.pref.vcore

<new_cpu_core_value_in_integer>

infagrid.orch.scheduler.oop.container.pref.memory

<new_memory_value_in_MB>

 

For instance, consider the following configurations, which are present in the connection, by default:

 

infagrid.orch.scheduler.oop.container.pref.vcore

4

infagrid.orch.scheduler.oop.container.pref.memory

5120

 

When additional 4GB is configured and CPU core requirement is reduced by '2', the new settings would be as follows:

 

infagrid.orch.scheduler.oop.container.pref.vcore

2

infagrid.orch.scheduler.oop.container.pref.memory

9216

 

infa_1021_hadoop_connection_blaze_properties.jpg 

 

       7. Save the changes made to the Hadoop connection. 

 

Pre-Informatica 10.2.1 versions:

 

  1. Login to Informatica Server machine.
  2. Navigate to '$INFA_HOME/services/shared/hadoop/[distribution]/infaConf' location.
  3. Update the 'hadoopEnv.properties' file, by adding 2-4G to the existing memory configuration (or) decreasing the CPU core configuration:​

  

infagrid.orch.scheduler.oop.container.pref.vcore=<new_cpu_core_value_in_integer>

infagrid.orch.scheduler.oop.container.pref.memory=<new_memory_value_in_MB>

 

For instance, consider the following configurations, which are present in the connection, by default:

 

infagrid.orch.scheduler.oop.container.pref.vcore=4

infagrid.orch.scheduler.oop.container.pref.memory=5120

 

When additional 4GB is configured and CPU core requirement is reduced by '2', the new settings would be as follows:

 

infagrid.orch.scheduler.oop.container.pref.vcore=2

infagrid.orch.scheduler.oop.container.pref.memory=9216

 

 

         4.    Once added, save the changes made to the file.

 

​Note: 

 

    • In case of multi-node setup, ensure that property is added to 'hadoopEnv.properties' file in all the nodes, primarily in the node where DIS used for mapping execution is running.
    • Property would be picked up automatically during mapping execution and it is not required to recycle the DIS for the changes to take effect.

 


After making the mentioned configuration changes, perform the following:


  • Stop the Blaze Engine if running. For information on stopping Blaze Engine, refer to the following KB article:

 

https://kb.informatica.com/howto/6/Pages/20/521162.aspx

 

  • Re-run the mapping in 'Blaze' mode and it should complete successfully. 
More Information
'infagrid.orch.scheduler.oop.container.pref.memory'  & 'infagrid.orch.scheduler.oop.container.pref.vcore'  are the configurations used by 'Blaze Engine' for the 'DTM' process, which would be performing the execution of submitted mappings in the Hadoop cluster. When the cluster has more YARN resources available, these settings should be configured based on the data volume to be processed by the mapping. 

  • When the volume of data processed by the submitted mappings to Blaze Engine is more, use more memory (oop.container.pref.memory) & fewer CPU cores (oop.container.pref.vcore).
  • When concurrency is important during the mapping execution, preference can be given to increase the 'CPU' cores (oop.container.pref.vcore) of Blaze Engine by proportionally increasing the memory (oop.container.pref.memory).​ 

For more information on 'Blaze Architecture', refer to the following document, which explains about 'Blaze Engine Architecture' and its core components:

 

https://docs.informatica.com/big-data-management/data-engineering-integration/10-4-0/administrator-guide/introduction-to-data-engineering-administration/hadoop-integration/run-time-process-on-the-blaze-engine.html

Applies To
Product: Data Engineering Integration(Big Data Management); Data Engineering Quality(Big Data Quality); Enterprise Data Preparation; Enterprise Data Catalog
Problem Type: Configuration; Performance; Crash/Hang; Stability
User Type: Administrator
Project Phase: Optimize; Implement
Product Version: Informatica 10.1; Informatica 10.1.1; HotFix; Informatica 10.2; Informatica 10.2.1; Informatica 10.2.1 Service Pack 1; Informatica 10.2.2; Informatica 10.4
Database:
Operating System:
Other Software:

Reference
Attachments
Last Modified Date:1/13/2020 11:36 PMID:567575
People who viewed this also viewed

Feedback

Did this KB document help you?



What can we do to improve this information (2000 or fewer characters)