scala - How to build Spark from the sources from the Download Spark page? -


i tried install , build spark 2.0.0 on ubuntu vm ubuntu 16.04 follows:

  1. install java

    sudo apt-add-repository ppa:webupd8team/java sudo apt-get update        sudo apt-get install oracle-java8-installer 
  2. install scala

    go downloads tab on site: scala-lang.org/download/all.html

    i used scala 2.11.8.

    sudo mkdir /usr/local/src/scala sudo tar -xvf scala-2.11.8.tgz -c /usr/local/src/scala/ 

    modify .bashrc file , include path scala:

    export scala_home=/usr/local/src/scala/scala-2.11.8 export path=$scala_home/bin:$path 

    then type:

    . .bashrc 
  3. install git

    sudo apt-get install git 
  4. download , build spark

    go to: http://spark.apache.org/downloads.html

    download spark 2.0.0 (build source - standalone mode).

    tar -xvf spark-2.0.0.tgz cd spark folder (that has been extracted). 

    now type:

    ./build/sbt assembly 

    after done installing, message:

    [success] total time: 1940 s, completed...

    followed date , time...

  5. run spark shell

    bin/spark-shell 

that's when hell breaks loose , start getting error. go assembly folder folder called target. there's no such folder there. things visible in assembly are: pom.xml, readme, , src.

i looked online quite while , haven't been able find single concrete solution solve error. can please provide explicit step-by-step instructions how go solving ?!? it's driving me nuts now... (t.t)

screenshot of error:

enter image description here

for reason, scala 2.11.8 not working while building if switch on scala 2.10.6 builds properly. guess reason need scala in first place access sbt able build spark. once built, need direct myself spark folder , type:

build/sbt package 

this build missing jar files me using scala 2.11... kinda weird that's how working (i assuming looking @ logs).

once spark builds again, type: bin/spark-shell (while being in spark folder) , you'll have access spark shell.


Comments