共计 9997 个字符，预计需要花费 25 分钟才能阅读完成。
欢送关注我的公众号 [极智视界]，回复 001 获取 Google 编程标准
O_o >_< o_O O_o ~_~ o_O
大家好，我是极智视界，本文分析一下 min-max 对称量化算法实现，以 Tengine 的实现为例。
Tengine 是 OpenAILab 开源的优良端侧深度学习推理框架，其外围次要由 C 语言实现，包裹的性能代码嵌套了 C++。量化是推理减速必不可少的优化环节，成熟的推理框架个别会把量化模块剥离进去造成独立的一套工具，如 Tengine、NCNN、昇腾、寒武纪都这么做，这次要是因为量化过程和硬件非强相干，解耦开来无能更多的事。
min-max 和 kl 量化算法是硬件厂商适配推理引擎的根底和标配，其中 kl 量化深受用户青睐，如英伟达的 TensorRT 也正是采纳了 kl 量化策略；而这里要介绍的 min-max 的特点是逻辑简略、成果良好，作为量化实现分享系列的开篇比拟适合，这里带大家一起钻研一下 Tengine 中 minx-max 量化策略的具体实现。
1、量化应用
量化次要分为激活值（动静）量化、权值 & 偏置（动态）量化，而权值 & 偏置的量化是对精度影响比拟大的，激活值的量化对整体影响较小，但也须要量化，才有可能协同达到整体称心的成果。对于个别量化来说，权值 & 偏置的量化会采纳逐通道 perChannel 的形式，而激活值的量化个别是逐层 perLayer 的形式。解释一下为啥会这样，对于量化来说，卷积必定是大头，对于卷积来说，若激活值量化采纳逐通道形式，这和卷积核参数共享是相悖的，所以个别激活值就用逐层量化，以符合卷积参数共享。
这里次要看一下 Tengine 量化须要的传参：
- Input model：传入的 fp32 tmfile 模型文件；
- Output model：生成的 int8 tmfile 模型文件；
- Calib images：传入的激活值量化校准图片；
- Scale file：生成的校准表文件；
- Agorithm：量化算法，可选 MIN-MAX、KL、ACIQ、DFQ、EQ；
- Dims：输出校准图的 shape，这里传三维 c h w，n 在代码中写死 n = 1；
- Mean：图像预处理均值；
- Scale：图像预处理缩放尺度；
- BGR2RGB：通道转换；
- Center crop：图像预处理，裁剪；
- Letter box：图像预处理，放弃横纵比的前提下对图像做 resize；
- YOLOv5 focus：相似 yolov5 的预处理注意力机制；
- Thread num：量化多线程设置；
2、min-max 量化
min-max 是最简略的量化算法，次要逻辑如下：
在 Tengine 中实现 min-max 办法的次要代码如下：
case ALGORITHM_MIN_MAX:{if (quant_tool.scale_file.empty()){
quant_tool.scale_file = "table_minmax.scale";
quant_tool.activation_quant_tool();}
save_graph_i8_perchannel(quant_tool.model_file.c_str(), quant_tool.scale_file.c_str(), quant_tool.output_file, quant_tool.inplace, false);
/* Evaluate quantitative losses */
if (quant_tool.evaluate){fprintf(stderr, "[Quant Tools Info]: Step Evaluate, evaluate quantitative losses\n");
quant_tool.assess_quant_loss(0);
}
break;
} 其中最次要的量化搜寻策略接口是 quant_tool.activation_quant_tool() 和 save_graph_i8_perchannel，对于 min-max 来说这两个接口别离做了两件事：
(1) 激活值量化，生成 table_minmax.scale；
(2) 权值 & 偏置量化，生成 scale_weight.txt 和 scale_bias.txt；
2.1 激活值量化
看 Tengine 源码肯定要抓住 struct graph* ir_graph，graph 这个构造体是精华。
激活值量化是个动静的过程，须要动静的获取每层的数据分布，这也就是为啥须要你喂肯定数量校准图片的起因。
先说一下预处理模块，这个其余量化算法是通用的：
// 将 input_tensor 和 input_data 地址绑定，而 input_tensor=>ir_graph->tensor_list。留神：这一步肯定要看到，不然后续代码很难看懂
tensor_t input_tensor = get_graph_input_tensor(ir_graph, 0, 0);
if (set_tensor_shape(input_tensor, dims, 4) < 0){fprintf(stderr, "Set input tensor shape failed\n");
return -1;
}
if (set_tensor_buffer(input_tensor, input_data.data(), img_size * sizeof(float)) < 0){fprintf(stderr, "Set input tensor buffer failed\n");
return -1;
}
// prerun graph，做一些初始化配置
if (prerun_graph_multithread(ir_graph, this->opt) < 0){fprintf(stderr, "Prerun multithread graph failed.\n");
return -1;
}
// 图像预处理，传出 input_data，这个和后面的 input_tensor & ir_graph->tensor_list[0] 输出参 绑定，批改了 input_data 即批改了 ir_graph.tensor_list，这样就能看懂
get_input_data_cv(imgs_list[nums].c_str(), input_data.data(), img_c, img_h, img_w, mean, scale, sw_RGB, center_crop, letterbox_rows, letterbox_cols, focus); 而后 run 一下，把两头激活值记录到 ir_graph->tensor_list[i] 里：
if (run_graph(ir_graph, 1) < 0){fprintf(stderr, "Run graph failed\n");
return -1;
}激活激活值的 min、max 值：
/* get the min/max value of activation tensor */
for (int i = 0; i < ir_graph->tensor_num; i++){struct tensor* act_tensor = ir_graph->tensor_list[i];
if (act_tensor->tensor_type == TENSOR_TYPE_VAR || act_tensor->tensor_type == TENSOR_TYPE_INPUT){float* start_addr = (float*)act_tensor->data;
float* end_addr = (float*)act_tensor->data + act_tensor->elem_num;
max_activation[i] = std::max(max_activation[i], *std::max_element(start_addr, end_addr));
min_activation[i] = std::min(min_activation[i], *std::min_element(start_addr, end_addr));}
} 计算激活值量化尺度，对于 softmax 层 scale 默认为 1 / 127.f：
/* save the calibration file with min-max algorithm */
FILE* fp_minmax = fopen("table_minmax.scale", "wb");
for (int i = 0; i < ir_graph->tensor_num; i++){struct tensor* t = ir_graph->tensor_list[i];
if (t->tensor_type == TENSOR_TYPE_VAR || t->tensor_type == TENSOR_TYPE_INPUT){
float act_scale = 1.f;
int act_zero_point = 0;
act_scale = std::max(std::abs(max_activation[i]), std::abs(min_activation[i])) / 127.f;
/* the scale of softmax is always scale = 1 / 127.f */
for (int j = 0; j < ir_graph->node_num; j++){struct node* noden = ir_graph->node_list[j];
struct tensor* tensor_tmp = get_ir_graph_tensor(ir_graph, noden->output_tensors[0]);
if (!(tensor_tmp->tensor_type == TENSOR_TYPE_INPUT || tensor_tmp->tensor_type == TENSOR_TYPE_VAR))
continue;
std::string tmp_op_name = get_op_name_from_type(noden->op.type);
std::string cur_name = t->name;
std::string tmp_name = tensor_tmp->name;
if ((cur_name == tmp_name) && tmp_op_name == "Softmax"){
act_scale = 1 / 127.f;
break;}
}
fprintf(fp_minmax, "%s %f %d\n", ir_graph->tensor_list[i]->name, act_scale, act_zero_point);}
}2.2 权值 & 偏置量化
权值 & 偏置量化和激活值量化不太一样，激活值量化须要校准图片推理以取得输出数据的动静散布，而权值 & 偏置是动态的，单纯的量化过程不需执行前向推理。
2.2.1 创立 graph
加载 tmfile，构建 graph：
struct graph* ir_graph = (struct graph*)create_graph(nullptr, "tengine", model_file);
if (nullptr == ir_graph){fprintf(stderr, "Create graph failed.\n");
return -1;}2.2.2 优化激活值量化 scale
这里次要做一个 quant.inplace 的优化，这是针对非卷积算子的量化解决策略。
if (inplace == 0){for (int i = 0; i < ir_graph->tensor_num; i++){struct tensor* ir_tensor = ir_graph->tensor_list[i];
if (ir_tensor->tensor_type == TENSOR_TYPE_VAR || ir_tensor->tensor_type == TENSOR_TYPE_INPUT){ir_tensor->scale = layer_scale[ir_tensor->name];
ir_tensor->zero_point = layer_zeropoint[ir_tensor->name];}}
}
else{
std::tr1::unordered_map<std::string, bool> layer_pass;
for (int i = ir_graph->tensor_num - 1; i >= 0; i--){struct tensor* ir_tensor = ir_graph->tensor_list[i];
if (ir_tensor->tensor_type == TENSOR_TYPE_VAR || ir_tensor->tensor_type == TENSOR_TYPE_INPUT){if (layer_pass[ir_tensor->name] == false){
uint32_t ir_node_idx = ir_tensor->producer;
struct node* t_node = ir_graph->node_list[ir_node_idx];
std::string op_name = get_op_name_from_type(t_node->op.type);
bool poolTrue = false;
bool reluTrue = false;
if (op_name == "Pooling"){struct pool_param* pool_param = (struct pool_param*)t_node->op.param_mem;
if (pool_param->pool_method == 0)
poolTrue = true;
}
else if (op_name == "ReLU"){struct relu_param* relu_param = (struct relu_param*)t_node->op.param_mem;
if (relu_param->negative_slope == 0.f)
reluTrue = true;
}
if (op_name == "Flatten" || op_name == "Reshape" || op_name == "Squeeze" || op_name == "Clip" || op_name == "Slice" || poolTrue || reluTrue){struct tensor* t_in_tensor = ir_graph->tensor_list[t_node->input_tensors[0]];
if (layer_scale[ir_tensor->name] != 0){ir_tensor->scale = layer_scale[ir_tensor->name];
ir_tensor->zero_point = layer_zeropoint[ir_tensor->name];
if (t_in_tensor->tensor_type == TENSOR_TYPE_VAR || t_in_tensor->tensor_type == TENSOR_TYPE_INPUT){recursion_pass_through(ir_graph, ir_tensor->name, t_in_tensor, layer_used, layer_scale, layer_zeropoint, layer_pass);}}
}
else{ir_tensor->scale = layer_scale[ir_tensor->name];
ir_tensor->zero_point = layer_zeropoint[ir_tensor->name];
}
layer_pass[ir_tensor->name] = true;}}}
}2.2.3 权值 & 偏置量化
量化的整个过程和激活值量化相似，即先搜寻 min、max 值，后做截断缩放解决。这里不仅须要计算 scale，而且还要做截断缩放解决的起因是须要生成 int8 tmfile 量化模型文件。这里还有一点须要留神的是权值量化精度为 int8，偏置量化精度为 int32，因为权值做完矩阵乘后值很有可能就会溢出 int8，所以须要权值矩阵乘后的值用 int32 存储，而后与 int32 的偏置做加法。
除了以上这些，和激活值量化还有个区别是，激活值量化是 perLayer 的，而权值 & 偏置量化是 perChannel 的。
权值 int8 量化：
/* quantize the weight data from fp32 to int8 */
if (op_name == "Convolution" || op_name == "FullyConnected" || op_name == "Deconvolution"){struct tensor* weight_tensor = ir_graph->tensor_list[noden->input_tensors[1]];
int channel_num = weight_tensor->dims[0];
int cstep = int(weight_tensor->elem_num / channel_num);
float* weight_data = (float*)weight_tensor->data;
int8_t* i8_weight_data = (int8_t*)sys_malloc(weight_tensor->elem_num * sizeof(int8_t));
float* weight_scale_list = (float*)sys_malloc(channel_num * sizeof(float));
int* weight_zp_list = (int*)sys_malloc(channel_num * sizeof(int));
fprintf(fp_weight, "%s", weight_tensor->name);
/* calculate the quant scale value of weight perchannel, scale = abs(min, max) / 127 */
if (internal){
// TODO
for (int ch = 0; ch < channel_num; ch++){weight_scale_list[ch] = weight_tensor->scale_list[ch];
weight_zp_list[ch] = 0;}
}
else{for (int ch = 0; ch < channel_num; ch++){
float* weight_data_ch_start = weight_data + ch * cstep;
float* weight_data_ch_end = weight_data + (ch + 1) * cstep;
float weight_max = *std::max_element(weight_data_ch_start, weight_data_ch_end);
float weight_min = *std::min_element(weight_data_ch_start, weight_data_ch_end);
weight_scale_list[ch] = std::max(std::abs(weight_max), std::abs(weight_min)) / 127.f;
weight_zp_list[ch] = 0;
fprintf(fp_weight, "%8.8f", weight_scale_list[ch]);
}
fprintf(fp_weight, "\n");
}
/* quantize the value of weight from Float32 to Int8, value_i8 = (value_fp32 / scale).round().clip(-127, 127) */
for (int ch = 0; ch < channel_num; ch++){for (int j = 0; j < cstep; j++){if (weight_data[ch * cstep + j] == 0 || weight_scale_list[ch] == 0)
i8_weight_data[ch * cstep + j] = 0;
else{float int8_data = round(weight_data[ch * cstep + j] / weight_scale_list[ch]);
int8_data = int8_data > 127.f ? 127.f : int8_data;
int8_data = int8_data < -127.f ? -127.f : int8_data;
i8_weight_data[ch * cstep + j] = int8_t(int8_data);}}
}
weight_tensor->scale_list = weight_scale_list;
weight_tensor->zp_list = weight_zp_list;
weight_tensor->data_type = TENGINE_DT_INT8;
weight_tensor->elem_size = sizeof(int8_t); // int8, signed char
weight_tensor->data = i8_weight_data;
weight_tensor->quant_param_num = channel_num;
}偏置 int32 量化：
/* quantize the weight data from fp32 to int32 */if (noden->input_num > 2){struct tensor* input_tensor = ir_graph->tensor_list[noden->input_tensors[0]]; struct tensor* bias_tensor = ir_graph->tensor_list[noden->input_tensors[2]]; float* bias_scale_list = (float*)sys_malloc(bias_tensor->dims[0] * sizeof(float)); int* bias_zp_list = (int*)sys_malloc(bias_tensor->dims[0] * sizeof(int32_t)); float* bias_data = (float*)bias_tensor->data; int* int32_bias_data = (int*)sys_malloc(bias_tensor->elem_num * sizeof(int32_t)); int bstep = int(bias_tensor->elem_num / channel_num); fprintf(fp_bias, "%s", bias_tensor->name); /* calculate the quant scale value of bias perchannel, scale = scale_weight * scale_in */ for (int ch = 0; ch < channel_num; ch++){bias_scale_list[ch] = weight_scale_list[ch] * input_tensor->scale; bias_zp_list[ch] = 0; fprintf(fp_bias, "%8.8f", bias_scale_list[ch]); } fprintf(fp_bias, "\n"); /* quantize the value of bias from Float32 to Int32, value_i32 = (value_fp32 / scale).round() */ for (int ch = 0; ch < channel_num; ch++){for (int bi = 0; bi < bstep; bi++){if (bias_data[ch * bstep + bi] == 0 || bias_scale_list[ch] == 0) int32_bias_data[ch * bstep + bi] = 0; else int32_bias_data[ch * bstep + bi] = int(round(bias_data[ch * bstep + bi] / bias_scale_list[ch]));} } bias_tensor->scale_list = bias_scale_list; bias_tensor->zp_list = bias_zp_list; bias_tensor->data_type = TENGINE_DT_INT32; bias_tensor->elem_size = sizeof(int32_t); // int32, signed int bias_tensor->data = int32_bias_data; bias_tensor->quant_param_num = channel_num;}到这里权值 & 偏置的量化就介绍的差不多咯。
以上具体介绍了 min-max 量化算法的实现，次要以 Tengine 为例进行代码阐明，心愿我的分享能对你的学习有一点帮忙。
【公众号传送】
《【模型推理】量化实现分享一：详解 min-max 对称量化算法实现》
正文完
