WHERE IS APACHE HIVE GOING? TO IN-MEMORY COMPUTING.-经管之家官网！

Let’s put this architecture to the test with a realistic dataset size and workload. Our previous performance blog, “Announcing Apache Hive 2.1: 25x Faster Queries and Much More”, discussed 4 reasons that LLAP delivers dramatically faster performance versus Hive on Tez. In that benchmark we saw 25+x performance boosts on ad-hoc queries with a dataset that fit entirely into the cluster’s memory.

In most cases, datasets will be far too large to fit in RAM so we need to understand if LLAP can truly tackle the big data challenge or if it’s limited to reporting roles on smaller datasets. To find out, we scaled the dataset up to 10 TB, 4x larger than aggregate cluster RAM, and we ran a number of far more complex queries.

Table 3 below shows how Hive LLAP is capable of running both At Speed and At Scale. The simplest query in the benchmark ran in 2.68 seconds on this 10 TB dataset while the most complex query, Query 64 performed a total of 37 joins and ran for more than 20 minutes.

http://hortonworks.com/wp-content/uploads/2016/10/Screen-Shot-2016-10-05-at-2.17.41-PM.png

If we look at the full results, we see substantial performance benefits across the board, with an average speedup of around 7x. The biggest winners are smaller-scale ad-hoc or star-schema-join type queries, but even extremely complex queries realized almost 2x benefit versus Hive 1.

http://hortonworks.com/wp-content/uploads/2016/10/Screen-Shot-2016-10-05-at-2.17.46-PM.png

If we focus just on smaller queries that run in 30 seconds or less we see about a 9x speedup. At this 10 TB scale a total of 24 queries run in 30 seconds or less with LLAP.

http://hortonworks.com/wp-content/uploads/2016/10/Screen-Shot-2016-10-05-at-8.35.20-PM.png

HDP TEST ENVIRONMENTSOFTWARE:

Hive 1 (HDP 2.5)	Hive 2 (HDP 2.5)
Hive 1.2.1 Tez 0.7.0 Hadoop 2.7.3 No LLAP	Hive 2.1.0 Tez 0.8.4 Hadoop 2.7.3 All queries run through LLAP

OTHER HIVE / HADOOP SETTINGS:

All HDP software was deployed using Apache Ambari using HDP 2.5 software. All defaults were used in our installation. In spite of that we mention relevant net-new settings in HDP 2.5 important in this benchmark. These optimizations are set by default for new HDP 2.5 installs. Upgraded HDP clusters would need to set these explicitly.

• hive.vectorized.execution.mapjoin.native.enabled=true;
• hive.vectorized.execution.mapjoin.native.fast.hashtable.enabled=true;
• hive.vectorized.execution.mapjoin.minmax.enabled=true;
• hive.vectorized.execution.reduce.enabled=true;
• hive.llap.client.consistent.splits=true;
• hive.optimize.dynamic.partition.hashjoin=true;
  

HARDWARE:

• 10x d2.8xlarge EC2 nodes
  

OS CONFIGURATION:

For the most part, OS defaults were used with 1 exception:

• /proc/sys/net/core/somaxconn = 4000
  

DATA:

• TPC-DS Scale 10000 data (10 TB), partitioned by date_sk columns, stored in ORC format with Zlib compression.
  

QUERIES:

• The Hive test was driven by the Hive Testbench https://github.com/hortonworks/hive-testbench/tree/hive14 in data generation and in queries. The same query text was used both for Hive 1 and for Hive 2.
  

扫码或添加微信号：坛友素质互助

「经管之家」APP：经管人学习、答疑、交友，就上经管之家！
免流量费下载资料----在经管之家app可以下载论坛上的所有资源，并且不额外收取下载高峰期的论坛币。
涵盖所有经管领域的优秀内容----覆盖经济、管理、金融投资、计量统计、数据分析、国贸、财会等专业的学习宝库，各类资料应有尽有。
来自五湖四海的经管达人----已经有上千万的经管人来到这里，你可以找到任何学科方向、有共同话题的朋友。
经管之家（原人大经济论坛），跨越高校的围墙，带你走进经管知识的新世界。
扫描下方二维码下载并注册APP