Wednesday, March 30, 2016

Run Spark on Windows 10 OS

In order for spark to run on Windows 10, we need to download the winutils.exe from and put it in the hadoop.bin directory.

To do this download the zip from and then put the files in the zip directory into the hadoop/bin directory (full path on my computer is C:\\hadoop\bin).

In your spark code, put this before invoking the JavaSparkContext:

        SparkConf conf = new SparkConf().setAppName("LeftJoin2");

        String home = System.getProperty("hadoop.home.dir");

        if (home == null) {
            System.setProperty("hadoop.home.dir", "C:\\hadoop");

        conf.setIfMissing("spark.master", "local[2]");

        JavaSparkContext context = new JavaSparkContext(conf);

No comments:

Post a Comment