乐趣区

关于hive:hivehadoop-配置项的调优Java-heap-space

hive 运行具体报错

WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = root_20201212183932_270ed399-a426-4f98-88d2-3c33c5433379
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Cannot run job locally: Input Size (= 94395226) is larger than hive.exec.mode.local.auto.inputbytes.max (= 50000000)
Starting Job = job_1607739316117_0009, Tracking URL = http://bigdata130:8088/proxy/application_1607739316117_0009/
Kill Command = /opt/server/hadoop-2.7.7/bin/hadoop job  -kill job_1607739316117_0009
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2020-12-12 18:39:35,991 Stage-1 map = 0%,  reduce = 0%
2020-12-12 18:39:50,312 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_1607739316117_0009 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1607739316117_0009_m_000000 (and more) from job job_1607739316117_0009

Task with the most failures(4): 
-----
Task ID:
  task_1607739316117_0009_m_000000

URL:
  http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1607739316117_0009&tipid=task_1607739316117_0009_m_000000
-----
Diagnostic Messages for this Task:
Error: Java heap space

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

解决方案

计划 1

设置 set io.sort.mb=10; 默认值是 100
io.sort.mb 的作用 排序所应用的内存数量。默认值:100M,须要与 mapred.child.java.opts 相配 默认:-Xmx200m。
不能超过 mapred.child.java.opt 设置,否则会 OOM。

Hadoop 作业 JVM 堆优化汇总及 JVM 复用

hive 执行 HQL 提醒出错 Error: Java heap space

hadoop 配置项的调优

退出移动版