Long Running Jobs in YARN distcp.

Usually distcp and other batch jobs run for a very long time. This is fine if you are running a HADOOP environment without kerberos. When we kerberize a cluster then we need to make sure we have keytabs/tokens renewed as they expire usually after 24hours.

In YARN we need to make sure it can auto renew the token once it reaches time limit. Below configuration is for cloudera setup but should work on any YARN environment.

1. YARN Service Update yarn-site.xml.

Under the ResourceManager Default Group > Advanced category, add the following string to the ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml property.

<property>
  <name>yarn.resourcemanager.proxy-user-privileges.enabled</name>
  <value>true</value>
</property>

2. HDFS Service Update core-site.xml.

<property>
  <name>hadoop.proxyuser.yarn.hosts</name>
  <value>*</value>
</property>

<property>
  <name>hadoop.proxyuser.yarn.groups</name>
  <value>*</value>
</property>

3. Restart YARN and HDFS.

NOTE: If you have other services then all services which depend on YARN and HDFS has to be restarted.

Useful Links.

• Cloudera
• Spark Long Running Jobs in YARN