Configuring Applications

You can override the default configurations for applications you install by supplying a configuration object when specifying applications you want installed at cluster creation time. Configuration objects consist of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file. You typically supply configurations in a list, allowing you to edit multiple configuration files in one JSON list.

Example JSON for a list of configurations is provided below:

[
  {
    "Classification": "core-site",
    "Properties": {
      "hadoop.security.groups.cache.secs": "250"
    }
  },
  {
    "Classification": "mapred-site",
    "Properties": {
      "mapred.tasktracker.map.tasks.maximum": "2",
      "mapreduce.map.sort.spill.percent": "90",
      "mapreduce.tasktracker.reduce.tasks.maximum": "5"
    }
  }
]

The classification usually specifies the file name that you want modified. An exception to this is the deprecated bootstrap action configure-daemons, which is used to set environment parameters such as --namenode-heap-size. Now, options like this are subsumed into the hadoop-env and yarn-env classifications with their own nested export classifications. Another exception is s3get, which was used to place a customer EncryptionMaterialsProvider object on each node in a cluster for use in client-side encryption. An option was added to the emrfs-site classification for this purpose.

An example of the hadoop-env classification is provided below:

[
  {
    "Classification": "hadoop-env",
    "Properties": {
      
    },
    "Configurations": [
      {
        "Classification": "export",
        "Properties": {
          "HADOOP_DATANODE_HEAPSIZE": "2048",
          "HADOOP_NAMENODE_OPTS": "-XX:GCTimeRatio=19"
        },
        "Configurations": [
          
        ]
      }
    ]
  }
]

An example of the yarn-env classification is provided below:

[
  {
    "Classification": "yarn-env",
    "Properties": {
      
    },
    "Configurations": [
      {
        "Classification": "export",
        "Properties": {
          "YARN_RESOURCEMANAGER_OPTS": "-Xdebug -Xrunjdwp:transport=dt_socket"
        },
        "Configurations": [
          
        ]
      }
    ]
  }
]

Bootstrap actions that were previously used to configure Hadoop and other applications are replaced by configurations. The following tables give classifications for components and applications and corollary bootstrap actions for each. If the classification matches a file documented in the application project, see that respective project documentation for more details.

The following settings do not belong to a configuration file but are used by Amazon EMR to potentially set multiple settings on your behalf.

aws emr create-cluster --release-label emr-4.0.0 --instance-type m3.xlarge --instance-count 2 --applications Name=Hive --configurations https://s3.amazonaws.com/mybucket/myfolder/myConfig.json

	Application hive = new Application();
		hive.withName("Hive");

	Map<String,String> hiveProperties = new HashMap<String,String>();
		hiveProperties.put("hive.join.emit.interval","1000");
		hiveProperties.put("hive.merge.mapfiles","true");
	    
	Configuration myHiveConfig = new Configuration()
		.withClassification("hive-site")
		.withProperties(hiveProperties);

	RunJobFlowRequest request = new RunJobFlowRequest()
		.withName("Create cluster with ReleaseLabel")
		.withReleaseLabel("emr-4.0.0")
		.withApplications(hive)
		.withConfigurations(myHiveConfig)
		.withServiceRole("EMR_DefaultRole")
		.withJobFlowRole("EMR_EC2_DefaultRole")
		.withInstances(new JobFlowInstancesConfig()
			.withEc2KeyName("myKey")
			.withInstanceCount(1)
			.withKeepJobFlowAliveWhenNoSteps(true)
			.withMasterInstanceType("m3.xlarge")
			.withSlaveInstanceType("m3.xlarge")
		);