Run Spark on Windows 10 OS

In order for spark to run on Windows 10, we need to download the winutils.exe from https://github.com/srccodes/hadoop-common-2.2.0-bin and put it in the hadoop.bin directory.

To do this download the zip from https://github.com/srccodes/hadoop-common-2.2.0-bin and then put the files in the zip directory into the hadoop/bin directory (full path on my computer is C:\\hadoop\bin).

In your spark code, put this before invoking the JavaSparkContext:

        SparkConf conf = new SparkConf().setAppName("LeftJoin2");

        String home = System.getProperty("hadoop.home.dir");

        if (home == null) {
            System.setProperty("hadoop.home.dir", "C:\\hadoop");
        }

        conf.setIfMissing("spark.master", "local[2]");

        JavaSparkContext context = new JavaSparkContext(conf);