2023 中国高校计算机大赛 — 大数据挑战赛：论文学科分类（清华大学主办）
官网地址：https://www.heywhale.com/home/competition
我的项目码源见文末
1. 较量介绍
- 赛事背景
自 2022 年底以来，大规模语言模型在各行各业产生了宽泛的利用，其中围绕学术工具开发也诞生了许多具备影响力的利用，例如 ChatPDF 等。另一方面，在 2023 年 3 月 14 日，智谱 AI 与清华大学联结公布了 ChatGLM-6B 开源模型，并在不到一个月的工夫内吸引了超过 100 万人下载安装。该模型在 Hugging Face (HF) 寰球大模型下载榜中间断 12 天位居第一名，在国内外的开源社区中产生了较大的影响。
为了最大化利用 ChatGLM-6B 开源模型推动科研工具的利用开发，咱们联合国内最具影响力的学术平台 AMiner，推出了本次「ChatGLM 实际大赛 · 学术利用篇」。本次较量的核心主题是如何利用 ChatGLM-6B 开源模型促成学术工具的优化。咱们心愿通过本次较量，为有志于投入大模型钻研和开发的爱好者提供一个实际平台。大赛共提供 3 个场景、7 个赛道，别离为：
-
场景 1：论文浏览
- 赛道一：论文学科分类 (Easy)——依据题目和摘要将论文精确分类到 40 个天然学科中去，可能单学科，也可能交叉学科，准确度达到 90% 以上。
- 赛道二：问答式科研知识库 (Medium)——将 PDF 论文上传构建向量化科研知识库，在知识库内做自在问答，要求绝对答复业余，且答案后要附带相干文件。
- 赛道三：论文综述和比照剖析 (Medium)——给定多篇论文的题目、摘要或全文，对论文的背景、问题、办法、试验、论断等进行综述或比照剖析。
-
场景 2：投稿审稿
- 赛道四：投稿期刊会议举荐 (Medium)——依据题目和摘要举荐适宜的 Top K 期刊会议，并依据匹配度针对每个举荐期刊会议给出举荐理由。
- 赛道五：审稿回复 (Medium)——基于 Openreview 数据，微调出一个审稿回复模型。
-
场景 3：论文发现
- 赛道六：论文检索 (Hard)——给定概念、给定问题、给定实体等的独自和混合检索。
- 赛道七：论文举荐和科技情报生成 (Hard)——基于用户画像（订阅关键词 + 搜寻浏览行为），从每日最新论文中筛选跟用户相干的 1 篇或多篇论文，基于论文信息（题目、作者、摘要等，也能够减少其余额定信息）微调大模型生成科技情报，情报模式和深度由选手自定义。
- 大赛组织
主办单位：智谱 AI
协办单位：和鲸科技
数据提供：AMiner 技术团队
组织反对：Huggingface
算力反对：揽睿星舟、AWS
2. 论文学科分类赛道工作简介
- 题目形容
依据题目和摘要将论文精确分类到 40 个天然学科里去，可能单学科，也可能交叉学科，准确度达到 90% 以上。
- 数据阐明
数据集：40 个天然学科下每个学科 500 篇论文的题目摘要，1000 篇左右交叉学科论文的题目和摘要。
测试集：500 篇文献，主观分类，评估指标 Acc。
局部原数据集展现：
{"id":155,"title":"Modeling heterogeneous network user route and departure time responses to dynamic pricing","abstract":"The ability to realistically capture trip-makers’responses to time-varying road charges is essential for network equilibrium assignment models typically applied to predict network flows in the presence of dynamic road (congestion) pricing. User responses to pricing are governed by individual trip-makers’preferences, such as their value of time (VOT), and the cost they attach to late vs. early arrival relative to the destination. These behavioral characteristics vary across users. This paper presents a joint route and departure time network equilibrium assignment model explicitly considering heterogeneous users with different preferred arrival times at destinations, VOT, and values of early and late schedule delays (VOESD and VOLSD). The model is formulated as an infinite-dimensional variational inequality and solved by a column generation-based algorithmic framework that embeds: (i) an extreme non-dominated alternative-generating algorithm to obtain combinations of VOT, VOESD, and VOLSD subintervals (or breakpoints) that define multiple user classes, and the corresponding least trip cost alternative (joint departure time and path) for each user class, (ii) a traffic simulator to capture traffic flow dynamics and determine experienced travel costs; and (iii) a multi-class alternative flow updating scheme to solve the reduced multi-class simultaneous route and departure time user equilibrium problem defined by a subset of feasible alternatives. Application to an actual network illustrates the properties of the algorithm, and underscores the importance of capturing user heterogeneity and temporal shifts in the appraisal of dynamic pricing schemes.","subject_name":["交通运输工程"]}
{"id":156,"title":"Duration-dependent effect of transient neonatal hypothyroidism on sertoli and germ cell number, and plasma and testicular interstitial fluid androgen binding protein concentration.","abstract":"The impact of transient neonatal hypothyroidism on growth and function of puberal testis during different milestones of postnatal testicular development was studied in Wister rats. Rat pups were made hypothyroid for 10, 15, 30, 40 and 60 days of postnatal age from birth by providing 0.05% (W\/V) methimazole (MMI) in the drinking water of the mother, from day 1 postpartum till weaning (25 days postpartum) and thereafter in the drinking water. Control rats were raised without MMI treatment. Sertoli cell number and its function was assessed on day 60 postpartum. Sertoli cell number increased consistently in 10, 15, 30 and 40 days transient hypothyroid rats but decreased in rats subjected to continuous hypothyroidism from birth to 60 days postpartum. Rats subjected to continuous hypothyroidism from birth showed spermatogenic arrest at puberty and had only a single layer of spermatogonia. Transient neonatal hypothyroidism for 10 (or) 15 days from birth increased spermatocytes (pachytene and zygotene), spermatids (elongated and round) whereas, that of 30 and 40 days decreases the number of germ cells. Plasma androgen binding protein (ABP) concentration decreased in puberal rats belonging to all groups, whereas the testicular interstitial fluid (TIF) concentration of ABP increased significantly in 10 and 15 days hypothyroid rats while it decreased in all other groups. These findings indicate that the mitogenic activity of Sertoli cell is increased irrespective of the duration of transient neonatal hypothyroidism. However, the functional activity of Sertoli cells (ABP production) in these puberal rats varies depending upon the postnatal period at which the animals were in hypothyroid state.","subject_name":["临床医学"]}
-
train.json 格局
{"id":0,"title":"title0","abstract":"abstract0","subject_name":["社会学"]} {"id":1,"title":"title1","abstract":"abstract1","subject_name":["社会学","石油工程"]} - test.json 预测文件格式（官网未放进去，我就简略结构几个作为测试）
{"id":0,"title":"title0","abstract":"abstract0",}
{"id":1,"title":"title1","abstract":"abstract1"}
{"id":0,"title":"Oxidative coupling of methane in the redox cyclic mode over the catalysts on the basis of CeO2 and La2O3","abstract":"The 1% CeO 2 , 9% La 2 O 3 \/SiO 2 and 2% CeO 2 , 8% La 2 O 3 \/SiO 2 catalysts show reliable efficiency in the OCM reaction, as well as stable work in the redox cyclic mode. Selectivity to C 2 products remarkably increases if preliminary reduction of the catalyst by a small amount of hydrogen is used."}
{"id":1,"title":"Tissue engineering: strategies, stem cells and scaffolds.","abstract":"Tissue engineering scaffolds are designed to influence the physical, chemical and biological environment surrounding a cell population. In this review we focus on our own work and introduce a range of strategies and materials used for tissue engineering, including the sources of cells suitable for tissue engineering: embryonic stem cells, bone marrow-derived mesenchymal stem cells and cord-derived mesenchymal stem cells. Furthermore, we emphasize the developments in custom scaffold design and manufacture, highlighting laser sintering, supercritical carbon dioxide processing, growth factor incorporation and zoning, plasma modification of scaffold surfaces, and novel multi-use temperature-sensitive injectable materials."}
{"id":2,"title":"Enhancement of Forced Convection Subcooled Film Boiling Heat Transfer Using Gas Sheet Collapse by Electric Field Application","abstract":"Enhancement of forced-convection boiling heat transfer by electriceld is investigated experimentally. When a high-temperature horizontallament is immersed in water, a gas sheet is formed around and the abovelament due to liquid boiling, in the early immersion process. This gas-sheet markedly decreases the boiling cooling rate of thelament. Here, forced collapse of the gas sheet is attempted by imposing an electriceld to enhance the boiling cooling rate, In the experiments, a horizontal platinum wire of 0.5mm in diameter is immersed in pure water under atmospheric pressure, and a DC voltage up to 600V is applied between the wire surface and an electrode made of glass placed 10mm apart. The whole boiling curve is measured under different applied voltages and wire-falling velocities in 0.5 to 2.0m\/s range, and at subcooling of 60 K. The experimental results show that the electric field is effective in promoting the disintegration of the gas sheet. Under the tested conditions, boiling cooling rate increased two-fold for an applied electriceld of 600 V\/cm. This result shows that the use of an electriceld to break up the gas-sheet has resulted in a remarkable increase in the cooling rate at high superheats during initial cooling period, which is even greater than that used in the existing material manufacturing processes by the rapid cooling method, and therefore, this method may contribute to developing new materials."}
{"id":195,"title":"Speciation of some heavy metals in bottom sediments of the Ob and Yenisei estuarine zones","abstract":"The speciation of Fe, Mn, Zn, Cu, Co, Ni, Cr, Pb, and Cd was studied in 52 samples of bottom sediments collected during Cruise 49 of the R\/V Dmitrii Mendeleev in estuaries of the Ob and Yenisei rivers in the southwestern Kara Sea. Immediately after sampling, the samples were subjected to on-board consecutive extraction to separate metal species according to their modes of occurrence in the sediments: (1) adsorbed, (2) amorphous Fe-Mn hydroxides and related metals, (3) organic + sulfide, and (4) residual, or lithogenic. The atomic absorption spectroscopy of the extracts was carried out at a stationary laboratory. The distribution of Fe, Zn, Cu, Co, Ni, Cr, Pb, and Cd species is characterized by the predominance of lithogenic or geochemically inert modes (70–95% of the bulk content), in which the metals are bound in terrigenous and clastic mineral particles and organic detritus. About half of the total Mn amount and 15–30% Zn and Cu is contained in geochemically mobile modes. The spatiotemporal variations in the proportions of metal species in the surface layer of sediments along the nearly meridional sections and through the vertical sections of bottom sediments cores testify that Mn and, to a lesser extent, Cu are the most sensitive to changes in the sedimentation environment. The role of their geochemically mobile species notably increases under reducing conditions."}3. 数据转换
将官网数据进行解决失去模型格局要求的输出，这边就间接给进去了，能够参考
其中 40 个分类别离为：
{'材料科学与工程', '临床医学', '电气工程', '数学', '化学', '地质工程', '地理学', '食品科学与工程', '医学', '生物学', '核科学与技术', '地球物理学', '水产', '药学', '交通运输工程', '体育学', '生物医学工程', '护理', '物理学', '心理学', '社会学', '神经科学', '计算机科学', '建筑学', '环境迷信与工程', '机械工程', '航空航天工程', '石油工程', '免疫与微生物学', '矿业', '通信与信息科学', '光学', '历史学', '地质学', '教育学', '海洋工程', '公共管理学', '仪器迷信与技术', '经济学', '音乐'}其中训练集和测试集比例 0.8：0.2
{"id": 11810, "text_a": "Restoration of the shear capacity for RC beams with web openings using precast SHCC plates Providing web opening in the shear-span zone of RC beams results in significant reduction in the shear capacity of such beams. Thus, an efficient restoration technique has to be found out and implemented in order to compensate the developed reduction. The main target of the current paper is to introduce and validate an innovative restoration technique for the new construction making use of the Strain-Hardening Cementitious Composites (SHCC) material. Accordingly, precast thin SHCC plates having the required opening were cast and cured for about 3 weeks to eliminate the volumetric change issues, and then placed inside the formwork at both sides before casting the RC beams included web openings. The chosen thickness of the SHCC plates was 20 mm in order to be easily accommodated in the concrete cover. For the considered openings, the opening depth was kept constant to be 0.30 of the beam effective depth, while the opening length was varied considering three values; 150, 300, and 450 mm. Besides, small amount of internal reinforcement in the form of steel wire mesh was provided inside some SHCC plates in order to enhance their shear strength and ductility. Experimental results showed that the provided SHCC layers enabled the strengthened beams to exhibit distinguished performance in terms of ultimate capacity, ductility and decreased shear crack width. In addition, the gain in shear capacity due to the SHCC plates is decreased with the increase of the opening width. Finally, comparisons between the obtained experimental results and the predicted shear capacities stipulated by the ACI 318-19 and JSCE codes were performed. The comparisons revealed that the estimated shear capacities are in satisfactory agreement with the experimental results, however, these estimations tend to be overestimated with the increase of the opening length.", "choices": ["交通运输工程", "体育学", "机械工程", "水产", "建筑学", "公共管理学", "医学", "地质学", "地球物理学", "生物学", "临床医学", "数学", "物理学", "化学", "石油工程", "历史学", "地质工程", "音乐", "核科学与技术", "护理", "经济学", "航空航天工程", "海洋工程", "社会学", "药学", "心理学", "矿业", "材料科学与工程", "电气工程", "教育学", "神经科学", "地理学", "光学", "环境迷信与工程", "计算机科学", "生物医学工程", "通信与信息科学", "免疫与微生物学", "食品科学与工程", "仪器迷信与技术"], "labels": [4]}
{"id": 5984, "text_a": "Caractéristiques et évaluation des symptômes de la rhinite allergique : Résultats de l’enquête CESAR Des recommandations sont publiées depuis plusieurs années pour la prise en charge de la rhinite allergique. Avec le temps, les concepts visant à définir les entités chroniques et celles qui se manifestent sur de plus courtes périodes ont évolué. Nous sommes ainsi passés du couple « perannuelle/saisonnière » à celui de « persistante/intermittente ». La sévérité des symptômes et leur répercussion sur la vie quotidienne des patients sont prises aussi en compte dans ces nouvelles recommandations. L’enquête observationnelle CESAR « Caractéristiques et Evaluation des Symptômes de la rhinite AlleRgique » vise à évaluer le paysage de la rhinite allergique en France sur ces nouveaux critères ainsi qu’à mieux appréhender les modalités de prise en charge des patients en médecine générale.", "choices": ["交通运输工程", "体育学", "机械工程", "水产", "建筑学", "公共管理学", "医学", "地质学", "地球物理学", "生物学", "临床医学", "数学", "物理学", "化学", "石油工程", "历史学", "地质工程", "音乐", "核科学与技术", "护理", "经济学", "航空航天工程", "海洋工程", "社会学", "药学", "心理学", "矿业", "材料科学与工程", "电气工程", "教育学", "神经科学", "地理学", "光学", "环境迷信与工程", "计算机科学", "生物医学工程", "通信与信息科学", "免疫与微生物学", "食品科学与工程", "仪器迷信与技术"], "labels": [10, 6]}
{"id": 4707, "text_a": "Dynamic analysis of OWT foundation with large diameter monopile under transient storm loading To investigate the stability of monopile foundation under dynamic loading, a comprehensive numerical model for the analysis of offshore wind turbines (OWT) foundation under a general transient storm loading is presented in this study. The dynamic stiffness and soil deformation around the large-diameter monopile is simulated using this method. During the numerical analysis, a dynamic boundary surface model of soil is derived instead of the empirical strength degradation. Along the axis direction of the monopile, an intensive study about deformation law of the seabed soil is analysed, moreover, some parameters which may affect the OWT stability and the dynamic stiffness are discussed. Some conclusions can be drawn that the dynamic stiffness and the lateral displacement of the monopile foundation can be obviously improved by increasing the buried depth than the diameter, and the proposed failure mode can well describe the failure law of soil around the monopile due to the dynamic loading.", "choices": ["交通运输工程", "体育学", "机械工程", "水产", "建筑学", "公共管理学", "医学", "地质学", "地球物理学", "生物学", "临床医学", "数学", "物理学", "化学", "石油工程", "历史学", "地质工程", "音乐", "核科学与技术", "护理", "经济学", "航空航天工程", "海洋工程", "社会学", "药学", "心理学", "矿业", "材料科学与工程", "电气工程", "教育学", "神经科学", "地理学", "光学", "环境迷信与工程", "计算机科学", "生物医学工程", "通信与信息科学", "免疫与微生物学", "食品科学与工程", "仪器迷信与技术"], "labels": [22]}
4. 模型训练预测
多任务训练场景可别离进行数据转换再进行混合：通用分类、评论情感剖析、语义类似度计算、蕴含推理、多项式浏览了解等泛滥“泛分类”工作
## 代码构造
├── deploy/simple_serving/ # 模型部署脚本
├── utils.py # 数据处理工具
├── run_train.py # 模型微调脚本
├── run_eval.py # 模型评估脚本
├── label_studio.py # 数据格式转换脚本
├── label_studio_text.md # 数据标注阐明文档
└── README.md4.1 模型微调
# 装置最新版本 paddlenlp
!pip install --upgrade paddlenlp# 挪动数据集
!cp /home/aistudio/input/train.txt /home/aistudio/data
!cp /home/aistudio/input/dev.txt /home/aistudio/data# 单卡启动：!python run_train.py \
--device gpu \
--logging_steps 100 \
--save_steps 100 \
--eval_steps 100 \
--seed 1000 \
--model_name_or_path utc-base \
--output_dir ./checkpoint/model_best \
--dataset_path ./data/ \
--max_seq_length 512 \
--per_device_train_batch_size 32 \
--per_device_eval_batch_size 32 \
--gradient_accumulation_steps 8 \
--num_train_epochs 20 \
--learning_rate 1e-5 \
--do_train \
--do_eval \
--do_export \
--export_model_dir ./checkpoint/model_best \
--overwrite_output_dir \
--disable_tqdm True \
--metric_for_best_model macro_f1 \
--load_best_model_at_end True \
--save_total_limit 1 \
--save_plm该示例代码中因为设置了参数 --do_eval，因而在训练完会主动进行评估。
可配置参数阐明：
single_label: 每条样本是否只预测一个标签。默认为False，示意多标签分类。device: 训练设施，可抉择 ‘cpu’、’gpu’ 其中的一种；默认为 GPU 训练。logging_steps: 训练过程中日志打印的距离 steps 数，默认 10。save_steps: 训练过程中保留模型 checkpoint 的距离 steps 数，默认 100。eval_steps: 训练过程中保留模型 checkpoint 的距离 steps 数，默认 100。seed：全局随机种子，默认为 42。model_name_or_path：进行 few shot 训练应用的预训练模型。默认为 “utc-base”, 可选 ”utc-xbase”, “utc-base”, “utc-medium”, “utc-mini”, “utc-micro”, “utc-nano”, “utc-pico”。output_dir：必须，模型训练或压缩后保留的模型目录；默认为None。dataset_path：数据集文件所在目录；默认为./data/。train_file：训练集后缀；默认为train.txt。dev_file：开发集后缀；默认为dev.txt。max_seq_len：文本最大切分长度，包含标签的输出超过最大长度时会对输出文本进行主动切分，标签局部不可切分，默认为 512。per_device_train_batch_size: 用于训练的每个 GPU 外围 /CPU 的 batch 大小，默认为 8。per_device_eval_batch_size: 用于评估的每个 GPU 外围 /CPU 的 batch 大小，默认为 8。num_train_epochs: 训练轮次，应用早停法时能够抉择 100；默认为 10。learning_rate：训练最大学习率，UTC 举荐设置为 1e-5；默认值为 3e-5。do_train: 是否进行微调训练，设置该参数示意进行微调训练，默认不设置。do_eval: 是否进行评估，设置该参数示意进行评估，默认不设置。do_export: 是否进行导出，设置该参数示意进行动态图导出，默认不设置。export_model_dir: 动态图导出地址，默认为 None。overwrite_output_dir：如果True，笼罩输入目录的内容。如果output_dir指向检查点目录，则应用它持续训练。disable_tqdm：是否应用 tqdm 进度条。metric_for_best_model：最优模型指标, UTC 举荐设置为macro_f1，默认为 None。load_best_model_at_end：训练完结后是否加载最优模型，通常与metric_for_best_model配合应用，默认为 False。save_total_limit：如果设置次参数，将限度 checkpoint 的总数。删除旧的 checkpoints输入目录，默认为 None。--save_plm：保留模型进行推理部署
NOTE:
如需复原模型训练，则能够设置 init_from_ckpt，如 init_from_ckpt=checkpoint/model_state.pdparams。
4.2 模型评估
通过运行以下命令进行模型评估预测：
# 评估样本
!python run_eval.py \
--model_path ./checkpoint/model_best \
--test_path ./data/dev.txt \
--per_device_eval_batch_size 32 \
--max_seq_len 512 \
--output_dir ./checkpoint_test 99%|██████████████████████████████████████████▌| 98/99 [00:31<00:00, 3.30it/s][2023-06-15 22:19:30,758] [INFO] - ***** test metrics *****
[2023-06-15 22:19:30,758] [INFO] - test_loss = 1.8884
[2023-06-15 22:19:30,758] [INFO] - test_macro_f1 = 0.8427
[2023-06-15 22:19:30,758] [INFO] - test_micro_f1 = 0.9849
[2023-06-15 22:19:30,759] [INFO] - test_runtime = 0:00:34.16
[2023-06-15 22:19:30,759] [INFO] - test_samples_per_second = 92.189
[2023-06-15 22:19:30,759] [INFO] - test_steps_per_second = 2.897
100%|███████████████████████████████████████████| 99/99 [00:33<00:00, 2.94it/s]可配置参数阐明：
model_path: 进行评估的模型文件夹门路，门路下需蕴含模型权重文件model_state.pdparams及配置文件model_config.json。test_path: 进行评估的测试集文件。per_device_eval_batch_size: 批处理大小，请联合机器状况进行调整，默认为 16。max_seq_len: 文本最大切分长度，输出超过最大长度时会对输出文本进行主动切分，默认为 512。single_label: 每条样本是否只预测一个标签。默认为False，示意多标签分类。
4.3 模型预测
paddlenlp.Taskflow装载定制模型，通过 task_path 指定模型权重文件的门路，门路下须要蕴含训练好的模型权重文件model_state.pdparams。
from pprint import pprint
import json
from paddlenlp import Taskflow
import pandas as pd
#读取文件并合并数据
data = []
ids = []
with open('/home/aistudio/input/test.json', 'r', encoding='utf-8') as f:
for line in f:
record = json.loads(line.strip())
text = record['title'] + '' + record['abstract']
data.append(text)
ids.append(record['id'])
schema = ['材料科学与工程', '临床医学', '电气工程', '数学', '化学', '地质工程', '地理学', '食品科学与工程', '医学', '生物学', '核科学与技术', '地球物理学', '水产', '药学', '交通运输工程', '体育学', '生物医学工程', '护理', '物理学', '心理学', '社会学', '神经科学', '计算机科学', '建筑学', '环境迷信与工程', '机械工程', '航空航天工程', '石油工程', '免疫与微生物学', '矿业', '通信与信息科学', '光学', '历史学', '地质学', '教育学', '海洋工程', '公共管理学', '仪器迷信与技术', '经济学', '音乐']
my_cls = Taskflow("zero_shot_text_classification", model="utc-base", schema=schema, task_path='/home/aistudio/checkpoint/model_best/plm')
results=my_cls(data)
#获取预测 labels
labels = []
for prediction in results:
label_list = []
for item in prediction['predictions']:
label_list.append(item['label'])
labels.append(label_list)
result = pd.DataFrame({'id': ids, 'subject_name': [labels[i] for i in range(len(labels))]})
print(result)
# 保留输入后果
result.to_csv('result.csv', index=False)
with open("/home/aistudio/output/output.txt", "w+",encoding='UTF-8') as f: #a : 写入文件，若文件不存在则会先创立再写入，但不会笼罩原文件，而是追加在文件开端
for result in results:
print(result)
line = json.dumps(result, ensure_ascii=False) #对中文默认应用的 ascii 编码. 想输入真正的中文须要指定 ensure_ascii=False
f.write(line + "\n")
print("数据后果已导出")[2023-06-16 14:58:56,470] [INFO] - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load 'utc-base'.
[2023-06-16 14:58:56,472] [INFO] - Already cached /home/aistudio/.paddlenlp/models/utc-base/utc_base_vocab.txt
[2023-06-16 14:58:56,495] [INFO] - tokenizer config file saved in /home/aistudio/.paddlenlp/models/utc-base/tokenizer_config.json
[2023-06-16 14:58:56,500] [INFO] - Special tokens file saved in /home/aistudio/.paddlenlp/models/utc-base/special_tokens_map.json
[2023-06-16 14:58:56,502] [INFO] - Assigning ['[O-MASK]'] to the additional_special_tokens key of the tokenizer
id subject_name
0 0 [化学]
1 1 [光学]
2 2 [物理学]
3 195 [化学]
{'predictions': [{'label': '化学', 'score': 0.9999739312596861}], 'text_a': 'Oxidative coupling of methane in the redox cyclic mode over the catalysts on the basis of CeO2 and La2O3 The 1% CeO 2 , 9% La 2 O 3 /SiO 2 and 2% CeO 2 , 8% La 2 O 3 /SiO 2 catalysts show reliable efficiency in the OCM reaction, as well as stable work in the redox cyclic mode. Selectivity to C 2 products remarkably increases if preliminary reduction of the catalyst by a small amount of hydrogen is used.'}
{'predictions': [{'label': '光学', 'score': 0.5380524000927461}], 'text_a': 'Tissue engineering: strategies, stem cells and scaffolds. Tissue engineering scaffolds are designed to influence the physical, chemical and biological environment surrounding a cell population. In this review we focus on our own work and introduce a range of strategies and materials used for tissue engineering, including the sources of cells suitable for tissue engineering: embryonic stem cells, bone marrow-derived mesenchymal stem cells and cord-derived mesenchymal stem cells. Furthermore, we emphasize the developments in custom scaffold design and manufacture, highlighting laser sintering, supercritical carbon dioxide processing, growth factor incorporation and zoning, plasma modification of scaffold surfaces, and novel multi-use temperature-sensitive injectable materials.'}
{'predictions': [{'label': '物理学', 'score': 0.8062627429802265}], 'text_a': 'Enhancement of Forced Convection Subcooled Film Boiling Heat Transfer Using Gas Sheet Collapse by Electric Field Application Enhancement of forced-convection boiling heat transfer by electriceld is investigated experimentally. When a high-temperature horizontallament is immersed in water, a gas sheet is formed around and the abovelament due to liquid boiling, in the early immersion process. This gas-sheet markedly decreases the boiling cooling rate of thelament. Here, forced collapse of the gas sheet is attempted by imposing an electriceld to enhance the boiling cooling rate, In the experiments, a horizontal platinum wire of 0.5mm in diameter is immersed in pure water under atmospheric pressure, and a DC voltage up to 600V is applied between the wire surface and an electrode made of glass placed 10mm apart. The whole boiling curve is measured under different applied voltages and wire-falling velocities in 0.5 t
{'predictions': [{'label': '化学', 'score': 0.6280942049702516}], 'text_a': 'Speciation of some heavy metals in bottom sediments of the Ob and Yenisei estuarine zones The speciation of Fe, Mn, Zn, Cu, Co, Ni, Cr, Pb, and Cd was studied in 52 samples of bottom sediments collected during Cruise 49 of the R/V Dmitrii Mendeleev in estuaries of the Ob and Yenisei rivers in the southwestern Kara Sea. Immediately after sampling, the samples were subjected to on-board consecutive extraction to separate metal species according to their modes of occurrence in the sediments: (1) adsorbed, (2) amorphous Fe-Mn hydroxides and related metals, (3) organic + sulfide, and (4) residual, or lithogenic. The atomic absorption spectroscopy of the extracts was carried out at a stationary laboratory. The distribution of Fe, Zn, Cu, Co, Ni, Cr, Pb, and Cd species is characterized by the predominance of lithogenic or geochemically inert modes (70–95% of the bulk content), in which the metals are bound in terrigeno
数据后果已导出
# 依照官网输入格局要求
import json
from paddlenlp import Taskflow
import pandas as pd
# 后盾将在 project 目录下运行，门路若不确定可写绝对路径 '/home/mw/project/xxx'
def invoke(input_data_path):
data = []
ids = []
with open('/home/aistudio/input/test.json', 'r', encoding='utf-8') as f:
for line in f:
record = json.loads(line.strip())
text = record['title'] + '' + record['abstract']
data.append(text)
ids.append(record['id'])
schema = ['材料科学与工程', '临床医学', '电气工程', '数学', '化学', '地质工程', '地理学', '食品科学与工程', '医学', '生物学', '核科学与技术', '地球物理学', '水产', '药学', '交通运输工程', '体育学', '生物医学工程', '护理', '物理学', '心理学', '社会学', '神经科学', '计算机科学', '建筑学', '环境迷信与工程', '机械工程', '航空航天工程', '石油工程', '免疫与微生物学', '矿业', '通信与信息科学', '光学', '历史学', '地质学', '教育学', '海洋工程', '公共管理学', '仪器迷信与技术', '经济学', '音乐']
my_cls = Taskflow("zero_shot_text_classification", model="utc-base", schema=schema, task_path='/home/aistudio/checkpoint/model_best/plm')
results=my_cls(data)
#pred_threshold 阈值函数记得批改
# 提取后果中的 label 值
#获取预测 labels
labels = []
for prediction in results:
label_list = []
for item in prediction['predictions']:
label_list.append(item['label'])
labels.append(label_list)
# 构建输入后果
result = pd.DataFrame({'id': ids, 'subject_name': [labels[i] for i in range(len(labels))]})
return result
input_data_path="/home/aistudio/input/test.json"
result=invoke(input_data_path)
print(result)[2023-06-16 14:59:12,503] [INFO] - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load 'utc-base'.
[2023-06-16 14:59:12,505] [INFO] - Already cached /home/aistudio/.paddlenlp/models/utc-base/utc_base_vocab.txt
[2023-06-16 14:59:12,528] [INFO] - tokenizer config file saved in /home/aistudio/.paddlenlp/models/utc-base/tokenizer_config.json
[2023-06-16 14:59:12,531] [INFO] - Special tokens file saved in /home/aistudio/.paddlenlp/models/utc-base/special_tokens_map.json
[2023-06-16 14:59:12,535] [INFO] - Assigning ['[O-MASK]'] to the additional_special_tokens key of the tokenizer
id subject_name
0 0 [化学]
1 1 [光学]
2 2 [物理学]
3 195 [化学]
5. 总结
赛道一：论文学科分类 (Easy)——依据题目和摘要将论文精确分类到 40 个天然学科中去，可能单学科，也可能交叉学科，准确度达到 90% 以上。整体工作比较简单，花了几个小时就搞完了，然而在官网镜像上节约好多工夫，导致工作提交失败，必须吐槽一下，提供的根底镜像只蕴含 TF 和 torch，没有 paddle，集体在构建新镜像始终公布不出导致婴儿惨死腹中，先把 baseline 开源进去，欢送大家调试，
5.1 改良策略
- 对摘要局部进行解决，做个文摘提取要害内容，目前模型字符解决长度 512，会失落局部信息。
我的项目码源见文末
我的项目云端码源链接链接：https://www.heywhale.com/mw/project/648c0e2e9de7b81463991943
更多优质内容请关注公号：汀丶人工智能