Skip Ribbon Commands
Skip to main content
Navigate Up
Sign In

Quick Launch

Average Rating:

facebook Twitter
Email
Print Bookmark Alert me when this article is updated

Feedback

HOW TO: Enable verbose class output for the Spark applications scheduled from Informatica BDM
Solution
Perform the following steps for enabling 'verbose' information related to 'Class loading' during the execution of Spark application in YARN, depending on the Informatica BDM version:

 

From Informatica 10.2.1 version:

 ​

  1.       .  Login to Informatica Administrator console or Informatica Developer Client tool.
  2.        Navigate to 'Connections'  tab in case of Admin console and 'Windows > Preferences > Connections > [Domain] > Cluster' , when Developer client is used.
  3.        Edit the 'Hadoop Pushdown Connection' being used for running Spark jobs.
  4.        Navigate to 'Spark Engine'  section in the connection.
  5.        Edit the Advanced Properties'  attribute in the section.
  6.        In the 'Advanced Properties' window, update the value of the property as follows:

 

spark.driver.extraJavaOptions

<existing_options> -verbose:class

spark.executor.extraJavaOptions

<existing_options> -verbose:class

1021_spark_advanced_prop_verbose_class_1.PNG

 

Note: Ensure that there is a space character between the end of the existing JVM argument (by default: -XX:MaxGCPauseMillis=500 ) and the verbose class argument '-verbose:class')

 

·       Once configured, save the changes made to the connection.

 

 

Pre-Informatica 10.2.1 versions:

 

 

1.    Login to Informatica Server machine.

2.    Navigate to '$INFA_HOME/services/shared/hadoop/[distribution]/infaConf' location.

3.    Update the 'hadoopEnv.properties' file for the following properties as follows:

 

infaspark.executor.extraJavaOptions=<existing_options> -verbose:class

infaspark.driver.cluster.mode.extraJavaOptions=<existing_options> -verbose:class

 

Note:

 

Ensure that there is a space character between the end of the existing JVM argument (by default: -XX:MaxMetaspaceSize=256M ) and the verbose class argument '-verbose:class')

 

              4.    Once added, save the changes made to the file.

 

Note:  

·       In case of multi-node setup, ensure that property is added to 'hadoopEnv.properties' file in all the nodes, primarily in the node, where DIS used for mapping execution is running.

·       Property would be picked up automatically during mapping execution and it is not required to recycle the DIS for the changes to take effect.

 

After making the mentioned configuration changes, on running new mappings in 'Spark' mode, 'verbose class loading' information would be available in the 'stderr' logs of the launched Spark YARN application. ​​

More Information
Reference
Applies To
Product: Data Engineering Integration(Big Data Management)
Problem Type: Configuration
User Type: Developer
Project Phase: Configure; Implement
Product Version:
Database:
Operating System:
Other Software:
Attachments
Last Modified Date:11/19/2019 4:28 AMID:604904
People who viewed this also viewed

Feedback

Did this KB document help you?



What can we do to improve this information (2000 or fewer characters)