scala-2.9.3:一种编程语言,下载地址:http://www.scala-lang.org/download/
spark-1.4.0:必须是编译好的Spark,如果下载的是Source,则需要自己根据环境使用SBT或者MAVEN重新编译才能使用。
编译好的 Spark下载地址:http://spark.apache.org/downloads.html。
2、安装scala-2.9.3#解压scala-2.9.3.tgztar -zxvf scala-2.9.3.tgz#配置SCALA_HOMEvi /etc/profile#添加如下环境export SCALA_HOME=/home/apps/scala-2.9.3export PATH=.:$SCALA_HOME/bin:$PATH#测试scala安装是否成功#直接输入scala3、安装spark-1.4.0
#解压spark-1.4.0.tgztar -zxvf spark-1.4.0.tgz#配置SPARK_HOMEvi /etc/profile#添加如下环境export SCALA_HOME=/home/apps/spark-1.4.0export PATH=.:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH
4、修改Spark配置文件#复制slaves.template和 spark-env.sh.template各一份cp spark-env.sh.template spark-env.shcp slaves.template slaves#slaves,此文件是指定子节点的主机,直接添加子节点主机名即可
在spark-env.sh末端添加如下几行:
#JDK安装路径export JAVA_HOME=/root/app/jdk#SCALA安装路径export SCALA_HOME=/root/app/scala-2.9.3#主节点的IP地址export SPARK_MASTER_IP=192.168.1.200#分配的内存大小export SPARK_WORKER_MEMORY=200m#指定hadoop的配置文件目录export HADOOP_CONF_DIR=/root/app/hadoop/etc/hadoop#指定worker工作时分配cpu数量export SPARK_WORKER_CORES=1#指定spark实例,一般1个足以export SPARK_WORKER_INSTANCES=1#jvm操作,在spark1.0之后增加了spark-defaults.conf默认配置文件,该配置参数在默认配置在该文件中export SPARK_JAVA_OPTSspark-defaults.conf中还有如下配置参数:
SPARK.MASTER //spark://hostname:8080SPARK.LOCAL.DIR //spark工作目录(做shuffle的目录)SPARK.EXECUTOR.MEMORY //spark1.0抛弃SPARK_MEM参数,使用该参数5、测试spark安装是否成功在主节点机器上启动顺序1、先启动hdfs(./sbin/start-dfs.sh)2、启动spark-master(./sbin/start-master.sh)3、启动spark-worker(./sbin/start-slaves.sh)4、jps查看进程有 主节点:namenode、secondrynamnode、master 从节点:datanode、worker5、启动spark-shell15/06/21 21:23:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable15/06/21 21:23:47 INFO spark.SecurityManager: Changing view acls to: root15/06/21 21:23:47 INFO spark.SecurityManager: Changing modify acls to: root15/06/21 21:23:47 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)15/06/21 21:23:47 INFO spark.HttpServer: Starting HTTP Server15/06/21 21:23:47 INFO server.Server: jetty-8.y.z-SNAPSHOT15/06/21 21:23:47 INFO server.AbstractConnector: Started SocketConnector@0 .0.0.0:3865115/06/21 21:23:47 INFO util.Utils: Successfully started service 'HTTP class server' on port 38651.Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.4.0 /_/ Using Scala version 2.10.4 (Java HotSpot(TM) Client VM, Java 1.7.0_65)Type in expressions to have them evaluated.Type :help for more information.15/06/21 21:23:54 INFO spark.SparkContext: Running Spark version 1.4.015/06/21 21:23:54 INFO spark.SecurityManager: Changing view acls to: root15/06/21 21:23:54 INFO spark.SecurityManager: Changing modify acls to: root15/06/21 21:23:54 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)15/06/21 21:23:56 INFO slf4j.Slf4jLogger: Slf4jLogger started15/06/21 21:23:56 INFO Remoting: Starting remoting15/06/21 21:23:57 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.200:57658]15/06/21 21:23:57 INFO util.Utils: Successfully started service 'sparkDriver' on port 57658.15/06/21 21:23:58 INFO spark.SparkEnv: Registering MapOutputTracker15/06/21 21:23:58 INFO spark.SparkEnv: Registering BlockManagerMaster15/06/21 21:23:58 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-4f1badf6-1e92-47ca-98a2-6d82f4882f15/blockmgr-530e4335-9e59-45d4-b9fb-6014089f5a0015/06/21 21:23:58 INFO storage.MemoryStore: MemoryStore started with capacity 267.3 MB15/06/21 21:23:59 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-4f1badf6-1e92-47ca-98a2-6d82f4882f15/httpd-4b2cca3c-e8d4-4ab3-9c3d-38ec579ec87315/06/21 21:23:59 INFO spark.HttpServer: Starting HTTP Server15/06/21 21:23:59 INFO server.Server: jetty-8.y.z-SNAPSHOT15/06/21 21:23:59 INFO server.AbstractConnector: Started SocketConnector@0 .0.0.0:5189915/06/21 21:23:59 INFO util.Utils: Successfully started service 'HTTP file server' on port 51899.15/06/21 21:23:59 INFO spark.SparkEnv: Registering OutputCommitCoordinator15/06/21 21:23:59 INFO server.Server: jetty-8.y.z-SNAPSHOT15/06/21 21:23:59 INFO server.AbstractConnector: Started SelectChannelConnector@0 .0.0.0:404015/06/21 21:23:59 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.15/06/21 21:23:59 INFO ui.SparkUI: Started SparkUI at
大数据分析师课程从数据分析、JAVA语言和linux操作系统入门知识入手,系统介绍Hadoop HDFS、MapReduce和Hbase等理论知识和hadoop的生态环境,详细演示hadoop三种模式的安装配置,重点讲解mahout+Spark大数据分析工具。
课程重点培养基于Hadoop架构的大数据分析思想及架构设计,掌握使用Hadoop架构应用于大数据分析过程。通过演示实际的大数据分析案例,使学员能在较短的时间内理解大数据分析的真实价值,提升成为兼有理论和实战的大数据分析师。
从课程体系设计和培训理念中,引导学员一步步深入学习,适合零基础但又有志于大数据行业的学员。
https://www.cda.cn/bigdata-jy.html