Implicit

一般的

implicit parameters

参考stackoverflow的帖子

  // probably in a library
class Prefixer(val prefix: String)
def addPrefix(s: String)(implicit p: Prefixer) = p.prefix + s

  // then probably in your application
implicit val myImplicitPrefixer = new Prefixer("***")
addPrefix("abc")  // returns "***abc"

如果定义了 implicit val那么就需要再传这个参数了

Implicit conversions

如果需要A，但是有B，那么在scope里就会去寻找 B=>A 的implicit value(或者去B和A的object里)。 也可以定义成 implicit def xyz(arg:B):A

implicit class

参见 官方文档

object Helpers {
  implicit class IntWithTimes(x: Int) {
    def times[A](f: => A): Unit = {
      def loop(current: Int): Unit =
        if(current > 0) {
          f
          loop(current - 1)
        }
      loop(x)
    }
  }
}


scala> import Helpers._
import Helpers._
scala> 5 times println("HI")

比如

object Job {
implicit class JobConverter(job: Job) {
    def toDoc(): Document = {
    }
}

val job:Job
//需要Document的地方我可以直接,而不需要把这个方法写在Job对象里
job.toDoc

Spark

Implicit Conversion

    val counts = words.map(word => (word, 1)).reduceByKey{case (x, y) => x + y}

reduceByKey需要import SparkContext对象才能用

import org.apache.spark.SparkContext._

reduceByKey 函数定义在PairRDDFunctions

class PairRDDFunctions[K, V](self: RDD[(K, V)])
    (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null)
  extends Logging
  with SparkHadoopMapReduceUtil
  with Serializable
{

  def reduceByKey(func: (V, V) => V): RDD[(K, V)] = {
    reduceByKey(defaultPartitioner(self), func)
  }

在对象SparkContext里定义了

object SparkContext extends Logging {

  implicit def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
      (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null) = {
    new PairRDDFunctions(rdd)
  }

因此最开始的代码会被编译器扩展成

val counts = SparkContext.rddToPairRDDFunctions(words.map(word => (word, 1))).reduceByKey{case (x, y) => x + y}

Implicit Parameters and ClassTag

上面PariRDDFunctioins的定义

根据
implicit TypeTag

implicit kt:ClassTag[K] 编译器会生成例如`kt= classTag(...)'的方法，则可以直接使用了

参见官方文档Using an Implicit Parameter of Type

Another sample

Spark SQL

import sqlContext.createSchemaRDD

    case class Person(name: String, age: Int)

    // Create an RDD of Person objects and register it as a table.
    val people = sc.textFile("src/main/resources/people.txt").map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt))
    people.registerTempTable("people")

registerTempTable定义在SchemaRDDLike里

def registerTempTable(tableName: String): Unit = {
    sqlContext.registerRDDAsTable(baseSchemaRDD, tableName)
  }

也就是说必须要是SchemaRDDLike类型对象才能掉这个方法，而people是一个RDD而已,具体说是MappedRDD 但是根据前面的createSchemaRDD, MappedRDD会被隐式转换为SchemaRDD

implicit def createSchemaRDD[A <: Product: TypeTag](rdd: RDD[A]) = {
    SparkPlan.currentContext.set(self)
    val attributeSeq = ScalaReflection.attributesFor[A]
    val schema = StructType.fromAttributes(attributeSeq)
    val rowRDD = RDDConversions.productToRowRdd(rdd, schema)
    new SchemaRDD(this, LogicalRDD(attributeSeq, rowRDD)(self))
  }