Create and Run Apache Spark Scala project using Eclipse and SBT

This post shows a simple way to create and run a apache spark scala project using eclipse and SBT. Following this post at link (http://czcodezone.blogspot.sg/2015/11/create-scala-project-using-scala-ide.html) to create a SBT-compatiable scala project in Scala IDE, open the build.sbt in the project and add the following line towards the end of the file.

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1"

Further more, since Spark needs to run a spark cluster in a jar, a sbt plugin must be added for the packaging. Create a file named "assembly.sbt" in the "project" folder under the root directory and put the following content:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

Now in the command line, navigate to the project root directory and run "sbt compile", "sbt package" command after you put in your spark scala code, then "sbt run" or "spark-submit" depending on whether you want to run it locally or submit to spark cluster

Create a Scala project using Scala-IDE and SBT on CentOS

This post shows how to create and build a Scala project using  Scala IDE and SBT (Simple Build Tool)

Step 1: Create  a Scala project using Scala IDE

Download, unzip and launch the Scala IDE (link:  http://scala-ide.org/), use the IDE to create a Scala project (said, with a name "ScalaHelloWorld")

Step 2: Install SBT

Download and install SBT using the following command:

> curl https://bintray.com/sbt/rpm/rpm | sudo tee /etc/yum.repos.d/bintray-sbt-rpm.repo
> sudo yum install sbt

Step 3: Configure Scala project for SBT

After installing the SBT, navigate to your "ScalaHelloWorld" root directory and create a build.sbt file in the directory with the following content:

name := "File Searcher"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.0" % "test"

Next in the root directory create a folder named "project" and create plugins.sbt file in the created "project" folder, with the following content:

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")

Now go back to the root directory and in the command line enter the following command:

> sbt eclipse

This will setup the eclipse scala project for sbt to work with. To begin writing Scala implementation, a simple recommendation with to create the folder structures like the following in the "src" folder under the root directory:

src/main/scala
src/main/resources
src/test/scala
src/test/resources

Now in the Scala IDE, right-click the folder "src/main/scala" and "src/test/scala" and select "Build Path -> Use as Source Folder", and just create your .scala file in these folders.

Step 4: Run SBT

After you code your scala implementation, navigate to the project root directory and type the following command to run the unit test codes:

> sbt test

And the following command to launch the app in main:

> sbt run