【Apache Spark】Redshift Data Source for Apache Spark

相似文件 换一批

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

Redshift Data Source for Apache Spark

本帖隐藏的内容

spark-redshift-master.zip (277.11 KB)

2017-4-18 04:25:48 上传

A library to load data into Spark SQL DataFrames from Amazon Redshift, and write them back to Redshift tables. Amazon S3 is used to efficiently transfer data in and out of Redshift, and JDBC is used to automatically trigger the appropriate COPY andUNLOAD commands on Redshift.

This library is more suited to ETL than interactive queries, since large amounts of data could be extracted to S3 for each query execution. If you plan to perform many queries against the same Redshift tables then we recommend saving the extracted data in a format such as Parquet.

• Installation
  • Snapshot builds
    
• Usage:
  • Data sources API: Scala, Python, SQL, R
  • Hadoop InputFormat
    
• Configuration
  • Authenticating to S3 and Redshift
  • Encryption
  • Parameters
    
• Additional configuration options
  • Configuring the maximum size of string columns
  • Setting a custom column type
  • Configuring column encoding
  • Setting descriptions on columns
    
• Transactional Guarantees
• Common problems and solutions
• S3 bucket and Redshift cluster are in different AWS regions
• Migration Guide
  

Installation

This library requires Apache Spark 2.0+ and Amazon Redshift 1.0.963+.

For version that works with Spark 1.x, please check for the 1.x branch.

You may use this library in your applications with the following dependency information:

Scala 2.10

groupId: com.databricksartifactId: spark-redshift_2.10version: 3.0.0-preview1

Scala 2.11

groupId: com.databricksartifactId: spark-redshift_2.11version: 3.0.0-preview1

You will also need to provide a JDBC driver that is compatible with Redshift. Amazon recommend that you use their driver, which is distributed as a JAR that is hosted on Amazon's website. This library has also been successfully tested using the Postgres JDBC driver.

Note on Hadoop versions: This library depends on spark-avro, which should automatically be downloaded because it is declared as a dependency. However, you may need to provide the corresponding avro-mapred dependency which matches your Hadoop distribution. In most deployments, however, this dependency will be automatically provided by your cluster's Spark assemblies and no additional action will be required.

Note on Amazon SDK dependency: This library declares a provided dependency on components of the AWS Java SDK. In most cases, these libraries will be provided by your deployment environment. However, if you get ClassNotFoundExceptions for Amazon SDK classes then you will need to add explicit dependencies on com.amazonaws.aws-java-sdk-core andcom.amazonaws.aws-java-sdk-s3 as part of your build / runtime configuration. See the comments inproject/SparkRedshiftBuild.scala for more details.

Snapshot builds

Master snapshot builds of this library are built using jitpack.io. In order to use these snapshots in your build, you'll need to add the JitPack repository to your build file.

• In Maven:

  <repositories>   <repository>     <id>jitpack.io</id>     <url>https://jitpack.io</url>   </repository></repositories>

  then

  <dependency>  <groupId>com.github.databricks</groupId>  <artifactId>spark-redshift_2.10</artifactId>  <!-- For Scala 2.11, use spark-redshift_2.11 instead -->  <version>master-SNAPSHOT</version></dependency>

• In SBT:

  resolvers += "jitpack" at "https://jitpack.io"

  then

  libraryDependencies += "com.github.databricks" %% "spark-redshift" % "master-SNAPSHOT"
  

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏1 回帖

关键词：Apache Spark apache Source Spark shift execution library tables write

谢谢楼主分享！