EMR Apache Tez内存调优

调优参考

https://community.cloudera.com/t5/Community-Articles/Demystify-Apache-Tez-Memory-Tuning-Step-by-Step/ta-p/245279

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_command-line-installation/content/determine-hdp-memory-config.html#calculate_yarn_mapreduce_memory_config

https://yijiyong.com/dp/yarnopt/01-basicparamopt.html

https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/performance-tuning/content/hive_tez_engine_properties.html

https://github.com/hortonworks/hdp-configuration-utils

执行脚本

[hadoop@10 ~/hdp-configuration-utils-master/2.1]$ python hdp-configuration-utils.py -c 16 -m 64 -d 11 -k False
 Using cores=16 memory=64GB disks=11 hbase=False
 Profile: cores=16 memory=57344MB reserved=8GB usableMem=56GB disks=11
 Num Container=20
 Container Ram=2560MB
 Used Ram=50GB
 Unused Ram=8GB
 ***** mapred-site.xml *****
 mapreduce.map.memory.mb=2560
 mapreduce.map.java.opts=-Xmx2048m
 mapreduce.reduce.memory.mb=2560
 mapreduce.reduce.java.opts=-Xmx2048m
 mapreduce.task.io.sort.mb=1024
 ***** yarn-site.xml *****
 yarn.scheduler.minimum-allocation-mb=2560
 yarn.scheduler.maximum-allocation-mb=51200
 yarn.nodemanager.resource.memory-mb=51200
 yarn.app.mapreduce.am.resource.mb=2560
 yarn.app.mapreduce.am.command-opts=-Xmx2048m
 ***** tez-site.xml *****
 tez.am.resource.memory.mb=2560
 tez.am.java.opts=-Xmx2048m
 ***** hive-site.xml *****
 hive.tez.container.size=2560
 hive.tez.java.opts=-Xmx2048m
 hive.auto.convert.join.noconditionaltask.size=671088000

机器规格

• 33台：3台core;30台task
• 16C64G, S6机型

SQL样例

select t1.chid
, t1.num_uv
, t2.num_new_uv
, t1.num_pv
, t1.num_hit_pv
, t1.num_hit_ratio
, t1.phone_pv
, t1.phone_hit_pv
, t1.phone_hit_ratio
from
(select manufacturer chid
, count(distinct imei) num_uv
, count(1) num_pv
, count(case when hit = '是' then 1 else null end ) as num_hit_pv
, count(case when hit = '是' then 1 else null end ) / count(1) num_hit_ratio
, count(distinct phone) phone_pv
, count(case when hit = '是' then phone else null end) as phone_hit_pv
, count(case when hit = '是' then phone else null end) / count(distinct phone) phone_hit_ratio
from ods.ods_num_sms_iccc_raw_log 
where dt = 20220529
and datatype<>1 
group by manufacturer
) t1

left join

(select
chid,
sum(case when f_time=20220519 then 1 else 0 end ) num_new_uv
from dwd.dwd_num_sms_iccc_user_info where dt=20220519 and datatype not in ('1','65535') and l_time='20220529' 
group by chid) t2

on t1.chid = t2.chid;

不同内存参数测试

2048M

内存参数

set mapreduce.map.memory.mb=2048;
set mapreduce.reduce.memory.mb=2048;
set mapreduce.task.io.sort.mb=768;
set yarn.scheduler.minimum-allocation-mb=2048;
set yarn.scheduler.maximum-allocation-mb=57344;
set yarn.nodemanager.resource.memory-mb=57344;
set yarn.app.mapreduce.am.resource.mb=2048;
set tez.am.resource.memory.mb=2048;
set hive.tez.container.size=2048;
set hive.auto.convert.join.noconditionaltask.size=402653000;
set tez.runtime.io.sort.mb=819;
set tez.runtime.unordered.output.buffer.size-mb=204;

资源利用率

vcores: 924, containers: 924 mem: 2T

结果

第一次

2560M

内存参数

set mapreduce.map.memory.mb=2560;
set mapreduce.reduce.memory.mb=2560;
set mapreduce.task.io.sort.mb=1024;
set yarn.scheduler.minimum-allocation-mb=2560;
set yarn.scheduler.maximum-allocation-mb=51200;
set yarn.nodemanager.resource.memory-mb=51200;
set yarn.app.mapreduce.am.resource.mb=2560;
set tez.am.resource.memory.mb=2560;
set hive.tez.container.size=2560;
set hive.auto.convert.join.noconditionaltask.size=671088000;
set tez.runtime.io.sort.mb=1024;
set tez.runtime.unordered.output.buffer.size-mb=256;

资源利用率

vcores: 759, containers: 759 mem: 2T

结果

第一次

第二次

3072M

内存参数

set mapreduce.map.memory.mb=3072;
set mapreduce.reduce.memory.mb=3072;
set mapreduce.task.io.sort.mb=1024;
set yarn.scheduler.minimum-allocation-mb=3072;
set yarn.scheduler.maximum-allocation-mb=55296;
set yarn.nodemanager.resource.memory-mb=55296;
set yarn.app.mapreduce.am.resource.mb=3072;
set tez.am.resource.memory.mb=3072;
set hive.tez.container.size=3072;
set hive.auto.convert.join.noconditionaltask.size=671088000;
set tez.runtime.io.sort.mb=1228;
set tez.runtime.unordered.output.buffer.size-mb=307;

资源利用率

vcores: 627, containers: 627 mem: 2T

结果

3584M

内存参数

set mapreduce.map.memory.mb=3584;
set mapreduce.reduce.memory.mb=3584;
set mapreduce.task.io.sort.mb=1280;
set yarn.scheduler.minimum-allocation-mb=3584;
set yarn.scheduler.maximum-allocation-mb=53760;
set yarn.nodemanager.resource.memory-mb=53760;
set yarn.app.mapreduce.am.resource.mb=3584;
set tez.am.resource.memory.mb=3584;
set hive.tez.container.size=3584;
set hive.auto.convert.join.noconditionaltask.size=805306000;
set tez.runtime.io.sort.mb=1433;
set tez.runtime.unordered.output.buffer.size-mb=358;

资源利用率

vcores: 528, containers: 528 mem: 2T

结果


4096M

内存参数

set mapreduce.map.memory.mb=4096;
set mapreduce.reduce.memory.mb=4096;
set mapreduce.task.io.sort.mb=1536;
set yarn.scheduler.minimum-allocation-mb=4096;
set yarn.scheduler.maximum-allocation-mb=53248;
set yarn.nodemanager.resource.memory-mb=53248;
set yarn.app.mapreduce.am.resource.mb=4096;
set tez.am.resource.memory.mb=4096;
set hive.tez.container.size=4096;
set hive.auto.convert.join.noconditionaltask.size=939524000;
set tez.runtime.io.sort.mb=1638;
set tez.runtime.unordered.output.buffer.size-mb=409;

资源利用率

vcores: 463, containers: 463 mem: 2T

结果

第一次

第二次

第三次

5120M

内存参数

set mapreduce.map.memory.mb=5120;
set mapreduce.reduce.memory.mb=5120;
set mapreduce.task.io.sort.mb=1792;
set yarn.scheduler.minimum-allocation-mb=5120;
set yarn.scheduler.maximum-allocation-mb=56320;
set yarn.nodemanager.resource.memory-mb=56320;
set yarn.app.mapreduce.am.resource.mb=5120;
set tez.am.resource.memory.mb=5120;
set hive.tez.container.size=5120;
set hive.auto.convert.join.noconditionaltask.size=1342177000;
set tez.runtime.io.sort.mb=2048;
set tez.runtime.unordered.output.buffer.size-mb=512;

资源利用率

vcores: 366, containers: 366 mem: 2T

结果

第一次

6144M

内存参数

set mapreduce.map.memory.mb=6144;
set mapreduce.reduce.memory.mb=6144;
set mapreduce.task.io.sort.mb=1792;
set yarn.scheduler.minimum-allocation-mb=6144;
set yarn.scheduler.maximum-allocation-mb=55296;
set yarn.nodemanager.resource.memory-mb=55296;
set yarn.app.mapreduce.am.resource.mb=6144;
set tez.am.resource.memory.mb=6144;
set hive.tez.container.size=6144;
set hive.auto.convert.join.noconditionaltask.size=1342177000;
set tez.runtime.io.sort.mb=2048;
set tez.runtime.unordered.output.buffer.size-mb=614;

资源利用率

vcores: 301, containers: 301 mem: 2T

结果

7168M

内存参数

set mapreduce.map.memory.mb=7168;
set mapreduce.reduce.memory.mb=7168;
set mapreduce.task.io.sort.mb=1792;
set yarn.scheduler.minimum-allocation-mb=7168;
set yarn.scheduler.maximum-allocation-mb=57344;
set yarn.nodemanager.resource.memory-mb=57344;
set yarn.app.mapreduce.am.resource.mb=7168;
set tez.am.resource.memory.mb=7168;
set hive.tez.container.size=6144;
set hive.auto.convert.join.noconditionaltask.size=1610612000;
set tez.runtime.io.sort.mb=2048;
set tez.runtime.unordered.output.buffer.size-mb=716;

资源利用率

vcores: 299, containers: 299 mem: 2T


结果

9216M

内存参数

set mapreduce.map.memory.mb=9216;
set mapreduce.reduce.memory.mb=6144;
set mapreduce.task.io.sort.mb=1792;
set yarn.scheduler.minimum-allocation-mb=6144;
set yarn.scheduler.maximum-allocation-mb=55296;
set yarn.nodemanager.resource.memory-mb=55296;
set yarn.app.mapreduce.am.resource.mb=6144;
set tez.am.resource.memory.mb=6144;
set hive.tez.container.size=6144;
set hive.auto.convert.join.noconditionaltask.size=2147483000;
set tez.runtime.io.sort.mb=2048;
set tez.runtime.unordered.output.buffer.size-mb=921;

资源利用率

vcores: 300, containers: 300 mem: 2T

结果

其他指标

COS

COS带宽限制：50Gb/s

监控按照5min进行统计。

计算每秒带宽公式=1.72x1024/5/60=5.87GBx8=46.9Gb

宿主机

S6机型带宽上限：13Gbps

监控按照1min进行统计。

计算每秒带宽公式=2496/1024x8=19.5Gbps，超过了带宽上限

高性能云盘

基准性能：6000 IOPS, 150 MB/s 带宽

带宽及IOPS未达到上限，磁盘IO繁忙比率偏高

结论

• 选择小容量容器执行速度更快

• 2048M～3072M执行速度接近，且执行时间最优

• 3584M～4096M执行速度接近

• 5120M以上执行速度接近

• 容器按照2056M基准分配内存