How to Run Spark Examples from IntelliJ

Here, I will explain how to run Apache Spark Application examples explained in this blog on windows using Scala & Maven from IntelliJ IDEA. Since the articles mentioned in this tutorial uses Apache Maven as the build system, we will use Maven to build the project.

Make sure you have the following before you proceed.

Clone Spark Examples GitHub Project into IntelliJ

Let’s clone Spark By Examples Github project into IntelliJ by using the Version Control option.

  • Open IntelliJ IDEA
  • Create a new project by selecting File > New > Project from Version Control.
Spark maven build intellij

Using this option, we are going to import the project directly from GitHub repository.

Spark maven build intellij
  • On Get from Version Control window, select the Version control as Git and enter the below Github URL for URL and enter the directory where you wanted to clone.

https://github.com/spark-examples/spark-scala-examples.git
  • If you don’t have Git installed, select the “Download and Install” option from the above window.
  • After Git installation, select the clone option which clones the project into your given folder.
  • This creates a new project on IntelliJ and starts cloning.
  • Now, wait for a few mins to complete the clone and also import the project into the workspace.

Once the cloning completes, you will see the following project workspace structure on IntelliJ.

Spark github clone

Run Maven build

Now run the Maven build. First, select the Maven from the right corner, navigate to Lifecycle > install, right-click, and select Run Maven Build.

Spark maven build intellij

This downloads all dependencies mentioned in the pom.xml file and compiles all examples in this tutorial. This also takes a few mins to complete and you should see the below message after a successful build.

Spark maven build intellij

Run Spark Program From IntelliJ

After successful Maven build, run src/main/scala/com.sparkbyexamples.spark.SparkSessionTest example from IntelliJ.

In case if you still get errors during the running of the Spark application, please restart the IntelliJ IDE and run the application again. Now you should see the below message in the console.

spark run example intelliJ

Once you complete the running Spark sample example in IntelliJ, you should read what is Spark Session, what is Spark Context, Spark RDD, Spark RDD Actions, Spark RDD Transformations.

You May Also Like Reading-

Happy Learning !!

Author: Shantun Parmar

4 thoughts on “How to Run Spark Examples from IntelliJ

  1. I am commenting to let you know of the fantastic encounter my cousin’s daughter experienced reading your site. She even learned many issues, not to mention what it’s like to have a wonderful giving mindset to have many people just grasp chosen grueling subject areas. You really exceeded our expectations. Many thanks for distributing the informative, healthy, educational and as well as easy thoughts on the topic to Jane.

  2. I’m also commenting to let you be aware of of the brilliant discovery my child encountered checking the blog. She came to understand a lot of details, including how it is like to have a marvelous coaching character to get folks completely fully grasp a variety of complex issues. You actually did more than her desires. I appreciate you for coming up with such essential, dependable, edifying and cool guidance on that topic to Ethel.

  3. I precisely had to say thanks all over again. I’m not certain the things I could possibly have created in the absence of the actual creative ideas discussed by you concerning such concern. It absolutely was a real depressing concern for me personally, but considering a new professional fashion you handled it forced me to cry for delight. I’m grateful for the assistance and as well , expect you comprehend what a powerful job that you are carrying out educating men and women by way of a web site. I am certain you’ve never come across all of us.

  4. I happen to be writing to make you understand what a wonderful experience our princess went through visiting yuor web blog. She realized too many things, with the inclusion of what it is like to possess an incredible teaching style to make the rest without problems understand various very confusing topics. You undoubtedly surpassed visitors’ expected results. Thanks for supplying those warm and helpful, trustworthy, revealing not to mention unique tips on your topic to Tanya.

Thanks for your support, You may click on ads to encourage us which assits to writers.

Leave a Reply

Your email address will not be published.