In order for spark to run on Windows 10, we need to download the winutils.exe from https://github.com/srccodes/hadoop-common-2.2.0-bin and put it in the hadoop.bin directory.
To do this download the zip from https://github.com/srccodes/hadoop-common-2.2.0-bin and then put the files in the zip directory into the hadoop/bin directory (full path on my computer is C:\\hadoop\bin).
In your spark code, put this before invoking the JavaSparkContext:
SparkConf conf = new SparkConf().setAppName("LeftJoin2");
String home = System.getProperty("hadoop.home.dir");
if (home == null) {
System.setProperty("hadoop.home.dir", "C:\\hadoop");
}
conf.setIfMissing("spark.master", "local[2]");
JavaSparkContext context = new JavaSparkContext(conf);
No comments:
Post a Comment