楼主: ReneeBK
1157 0

[Case Study]Random Forest using Scala [推广有奖]

  • 1关注
  • 62粉丝

VIP

学术权威

14%

还不是VIP/贵宾

-

TA的文库  其他...

R资源总汇

Panel Data Analysis

Experimental Design

0%
威望
1 级
论坛币
49422 个
通用积分
52.2304
学术水平
370 点
热心指数
273 点
信用等级
335 点
经验
57815 点
帖子
4006
精华
21
在线时间
582 小时
注册时间
2005-5-8
最后登录
2023-11-26
楼主
ReneeBK 发表于 2015-11-16 00:44:12 |只看作者 |坛友微信交流群|倒序 |AI写论文

+2 论坛币
k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群
赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币
  1. // scalastyle:off println
  2. package org.apache.spark.examples.ml

  4. import scala.collection.mutable
  5. import scala.language.reflectiveCalls

  7. import scopt.OptionParser

  9. import org.apache.spark.{SparkConf, SparkContext}
  10. import org.apache.spark.examples.mllib.AbstractParams
  11. import org.apache.spark.ml.{Pipeline, PipelineStage}
  12. import org.apache.spark.ml.classification.{RandomForestClassificationModel, RandomForestClassifier}
  13. import org.apache.spark.ml.feature.{StringIndexer, VectorIndexer}
  14. import org.apache.spark.ml.regression.{RandomForestRegressionModel, RandomForestRegressor}
  15. import org.apache.spark.sql.DataFrame


  18. /**
  19. * An example runner for decision trees. Run with
  20. * {{{
  21. * ./bin/run-example ml.RandomForestExample [options]
  22. * }}}
  23. * Decision Trees and ensembles can take a large amount of memory.  If the run-example command
  24. * above fails, try running via spark-submit and specifying the amount of memory as at least 1g.
  25. * For local mode, run
  26. * {{{
  27. * ./bin/spark-submit --class org.apache.spark.examples.ml.RandomForestExample --driver-memory 1g
  28. *   [examples JAR path] [options]
  29. * }}}
  30. * If you use it as a template to create your own app, please use `spark-submit` to submit your app.
  31. */
  32. object RandomForestExample {

  34.   case class Params(
  35.       input: String = null,
  36.       testInput: String = "",
  37.       dataFormat: String = "libsvm",
  38.       algo: String = "classification",
  39.       maxDepth: Int = 5,
  40.       maxBins: Int = 32,
  41.       minInstancesPerNode: Int = 1,
  42.       minInfoGain: Double = 0.0,
  43.       numTrees: Int = 10,
  44.       featureSubsetStrategy: String = "auto",
  45.       fracTest: Double = 0.2,
  46.       cacheNodeIds: Boolean = false,
  47.       checkpointDir: Option[String] = None,
  48.       checkpointInterval: Int = 10) extends AbstractParams[Params]

  50.   def main(args: Array[String]) {
  51.     val defaultParams = Params()

  53.     val parser = new OptionParser[Params]("RandomForestExample") {
  54.       head("RandomForestExample: an example random forest app.")
  55.       opt[String]("algo")
  56.         .text(s"algorithm (classification, regression), default: ${defaultParams.algo}")
  57.         .action((x, c) => c.copy(algo = x))
  58.       opt[Int]("maxDepth")
  59.         .text(s"max depth of the tree, default: ${defaultParams.maxDepth}")
  60.         .action((x, c) => c.copy(maxDepth = x))
  61.       opt[Int]("maxBins")
  62.         .text(s"max number of bins, default: ${defaultParams.maxBins}")
  63.         .action((x, c) => c.copy(maxBins = x))
  64.       opt[Int]("minInstancesPerNode")
  65.         .text(s"min number of instances required at child nodes to create the parent split," +
  66.         s" default: ${defaultParams.minInstancesPerNode}")
  67.         .action((x, c) => c.copy(minInstancesPerNode = x))
  68.       opt[Double]("minInfoGain")
  69.         .text(s"min info gain required to create a split, default: ${defaultParams.minInfoGain}")
  70.         .action((x, c) => c.copy(minInfoGain = x))
  71.       opt[Int]("numTrees")
  72.         .text(s"number of trees in ensemble, default: ${defaultParams.numTrees}")
  73.         .action((x, c) => c.copy(numTrees = x))
  74.       opt[String]("featureSubsetStrategy")
  75.         .text(s"number of features to use per node (supported:" +
  76.         s" ${RandomForestClassifier.supportedFeatureSubsetStrategies.mkString(",")})," +
  77.         s" default: ${defaultParams.numTrees}")
  78.         .action((x, c) => c.copy(featureSubsetStrategy = x))
  79.       opt[Double]("fracTest")
  80.         .text(s"fraction of data to hold out for testing.  If given option testInput, " +
  81.         s"this option is ignored. default: ${defaultParams.fracTest}")
  82.         .action((x, c) => c.copy(fracTest = x))
  83.       opt[Boolean]("cacheNodeIds")
  84.         .text(s"whether to use node Id cache during training, " +
  85.         s"default: ${defaultParams.cacheNodeIds}")
  86.         .action((x, c) => c.copy(cacheNodeIds = x))
  87.       opt[String]("checkpointDir")
  88.         .text(s"checkpoint directory where intermediate node Id caches will be stored, " +
  89.         s"default: ${
  90.           defaultParams.checkpointDir match {
  91.             case Some(strVal) => strVal
  92.             case None => "None"
  93.           }
  94.         }")
  95.         .action((x, c) => c.copy(checkpointDir = Some(x)))
  96.       opt[Int]("checkpointInterval")
  97.         .text(s"how often to checkpoint the node Id cache, " +
  98.         s"default: ${defaultParams.checkpointInterval}")
  99.         .action((x, c) => c.copy(checkpointInterval = x))
  100.       opt[String]("testInput")
  101.         .text(s"input path to test dataset.  If given, option fracTest is ignored." +
  102.         s" default: ${defaultParams.testInput}")
  103.         .action((x, c) => c.copy(testInput = x))
  104.       opt[String]("dataFormat")
  105.         .text("data format: libsvm (default), dense (deprecated in Spark v1.1)")
  106.         .action((x, c) => c.copy(dataFormat = x))
  107.       arg[String]("<input>")
  108.         .text("input path to labeled examples")
  109.         .required()
  110.         .action((x, c) => c.copy(input = x))
  111.       checkConfig { params =>
  112.         if (params.fracTest < 0 || params.fracTest >= 1) {
  113.           failure(s"fracTest ${params.fracTest} value incorrect; should be in [0,1).")
  114.         } else {
  115.           success
  116.         }
  117.       }
  118.     }

  120.     parser.parse(args, defaultParams).map { params =>
  121.       run(params)
  122.     }.getOrElse {
  123.       sys.exit(1)
  124.     }
  125.   }

  127.   def run(params: Params) {
  128.     val conf = new SparkConf().setAppName(s"RandomForestExample with $params")
  129.     val sc = new SparkContext(conf)
  130.     params.checkpointDir.foreach(sc.setCheckpointDir)
  131.     val algo = params.algo.toLowerCase

  133.     println(s"RandomForestExample with parameters:\n$params")

  135.     // Load training and test data and cache it.
  136.     val (training: DataFrame, test: DataFrame) = DecisionTreeExample.loadDatasets(sc, params.input,
  137.       params.dataFormat, params.testInput, algo, params.fracTest)

  139.     // Set up Pipeline
  140.     val stages = new mutable.ArrayBuffer[PipelineStage]()
  141.     // (1) For classification, re-index classes.
  142.     val labelColName = if (algo == "classification") "indexedLabel" else "label"
  143.     if (algo == "classification") {
  144.       val labelIndexer = new StringIndexer()
  145.         .setInputCol("label")
  146.         .setOutputCol(labelColName)
  147.       stages += labelIndexer
  148.     }
  149.     // (2) Identify categorical features using VectorIndexer.
  150.     //     Features with more than maxCategories values will be treated as continuous.
  151.     val featuresIndexer = new VectorIndexer()
  152.       .setInputCol("features")
  153.       .setOutputCol("indexedFeatures")
  154.       .setMaxCategories(10)
  155.     stages += featuresIndexer
  156.     // (3) Learn Random Forest
  157.     val dt = algo match {
  158.       case "classification" =>
  159.         new RandomForestClassifier()
  160.           .setFeaturesCol("indexedFeatures")
  161.           .setLabelCol(labelColName)
  162.           .setMaxDepth(params.maxDepth)
  163.           .setMaxBins(params.maxBins)
  164.           .setMinInstancesPerNode(params.minInstancesPerNode)
  165.           .setMinInfoGain(params.minInfoGain)
  166.           .setCacheNodeIds(params.cacheNodeIds)
  167.           .setCheckpointInterval(params.checkpointInterval)
  168.           .setFeatureSubsetStrategy(params.featureSubsetStrategy)
  169.           .setNumTrees(params.numTrees)
  170.       case "regression" =>
  171.         new RandomForestRegressor()
  172.           .setFeaturesCol("indexedFeatures")
  173.           .setLabelCol(labelColName)
  174.           .setMaxDepth(params.maxDepth)
  175.           .setMaxBins(params.maxBins)
  176.           .setMinInstancesPerNode(params.minInstancesPerNode)
  177.           .setMinInfoGain(params.minInfoGain)
  178.           .setCacheNodeIds(params.cacheNodeIds)
  179.           .setCheckpointInterval(params.checkpointInterval)
  180.           .setFeatureSubsetStrategy(params.featureSubsetStrategy)
  181.           .setNumTrees(params.numTrees)
  182.       case _ => throw new IllegalArgumentException("Algo ${params.algo} not supported.")
  183.     }
  184.     stages += dt
  185.     val pipeline = new Pipeline().setStages(stages.toArray)

  187.     // Fit the Pipeline
  188.     val startTime = System.nanoTime()
  189.     val pipelineModel = pipeline.fit(training)
  190.     val elapsedTime = (System.nanoTime() - startTime) / 1e9
  191.     println(s"Training time: $elapsedTime seconds")

  193.     // Get the trained Random Forest from the fitted PipelineModel
  194.     algo match {
  195.       case "classification" =>
  196.         val rfModel = pipelineModel.stages.last.asInstanceOf[RandomForestClassificationModel]
  197.         if (rfModel.totalNumNodes < 30) {
  198.           println(rfModel.toDebugString) // Print full model.
  199.         } else {
  200.           println(rfModel) // Print model summary.
  201.         }
  202.       case "regression" =>
  203.         val rfModel = pipelineModel.stages.last.asInstanceOf[RandomForestRegressionModel]
  204.         if (rfModel.totalNumNodes < 30) {
  205.           println(rfModel.toDebugString) // Print full model.
  206.         } else {
  207.           println(rfModel) // Print model summary.
  208.         }
  209.       case _ => throw new IllegalArgumentException("Algo ${params.algo} not supported.")
  210.     }

  212.     // Evaluate model on training, test data
  213.     algo match {
  214.       case "classification" =>
  215.         println("Training data results:")
  216.         DecisionTreeExample.evaluateClassificationModel(pipelineModel, training, labelColName)
  217.         println("Test data results:")
  218.         DecisionTreeExample.evaluateClassificationModel(pipelineModel, test, labelColName)
  219.       case "regression" =>
  220.         println("Training data results:")
  221.         DecisionTreeExample.evaluateRegressionModel(pipelineModel, training, labelColName)
  222.         println("Test data results:")
  223.         DecisionTreeExample.evaluateRegressionModel(pipelineModel, test, labelColName)
  224.       case _ =>
  225.         throw new IllegalArgumentException("Algo ${params.algo} not supported.")
  226.     }

  228.     sc.stop()
  229.   }
  230. }
  231. // scalastyle:on println
复制代码


二维码

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0
关键词：Case study Forest random study Using Random import

相关帖子

本帖被以下文库推荐

使用道具

返回列表
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 我要注册

本版微信群
加JingGuanBbs
拉您进交流群
  • 手机版  |
  • 意见反馈  |
  • 帮助  |
  • 新手入门  |
  • 用户手册  |
  • 友情链接  |

    • 如有投资本站、合作意向或投放广告，请联系：13661292478（刘老师）

    联系客服

    邮箱：service@pinggu.org 投诉或不良信息处理：（010-68466864）

    010-68466864

    京ICP备16021002-2号 京B2-20170662号 京公网安备 11010802022788号 论坛法律顾问：王进律师 知识产权保护声明   免责及隐私声明

    GMT+8, 2024-5-29 11:19