bug_fix (#3806)

* update link

* rename demo to examples

* add python readme

* update the cmakelist

* add ernie python demo

* add status print after compiling

* Add README for ernie fast tokenizer

* Add clip fast tokenizer cpp readme

* Update docs

* Add some tips

* add deps

* add deps

* update

* update

* update shell

README_cn.md

@@ -30,6 +30,9 @@
 **PaddleNLP**是一款**简单易用**且**功能强大**的自然语言处理开发库。聚合业界**优质预训练模型**并提供**开箱即用**的开发体验，覆盖NLP多场景的模型库搭配**产业实践范例**可满足开发者**灵活定制**的需求。
 
 ## News 📢
+
+* 🔥 **2022.11.12 PaddleNLP新增AutoPrompt自动化提示功能，登顶FewCLUE小样本学习榜单!**
+  * 🥇 PaddleNLP 团队开源了 **AutoPrompt** 方案，基于开源的文心 ERNIE 预训练语言模型 ，结合了领域预训练和自动化提示学习技术，以291M 参数量的模型在小样本权威学习榜单 FewCLUE 排名第一，[详见](https://mp.weixin.qq.com/s/_JPiAzFA1f0BZ0igdv-EKA)。
 * 🔥 **2022.10.27 发布 [PaddleNLP v2.4.2](https://github.com/PaddlePaddle/PaddleNLP/releases/tag/v2.4.2)**
   * NLG能力扩充：新增📄[**基于Pegasus的中文文本摘要方案**](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/applications/text_summarization/pegasus)，效果领先；新增❓[**问题生成解决方案**](./examples/question_generation)，提供基于业界领先模型UNIMO-Text和大规模多领域问题生成数据集训练的通用问题生成预训练模型。均支持Taskflow一键调用，支持FasterGeneration高性能推理，训练推理部署全流程打通。
   * 发布 🖼[**PPDiffusers**](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/ppdiffusers)：支持跨模态（如图像与语音）训练和推理的扩散模型（Diffusion Model）工具箱，可快速体验、二次开发 **Stable Diffusion**，持续支持更多模型。
@@ -234,17 +237,17 @@ PaddleNLP针对信息抽取、语义检索、智能问答、情感分析等高
 
 ### 高性能分布式训练与推理
 
-#### ⚡ FasterTokenizer：高性能文本处理库
+#### ⚡ FastTokenizer：高性能文本处理库
 
 <div align="center">
     <img src="https://user-images.githubusercontent.com/11793384/168407921-b4395b1d-44bd-41a0-8c58-923ba2b703ef.png" width="400">
 </div>
 
 ```python
-AutoTokenizer.from_pretrained("ernie-3.0-medium-zh", use_faster=True)
+AutoTokenizer.from_pretrained("ernie-3.0-medium-zh", use_fast=True)
 ```
 
-为了实现更极致的模型部署性能，安装FastTokenizers后只需在`AutoTokenizer` API上打开 `use_faster=True`选项，即可调用C++实现的高性能分词算子，轻松获得超Python百余倍的文本处理加速，更多使用说明可参考[FasterTokenizer文档](./faster_tokenizer)。
+为了实现更极致的模型部署性能，安装FastTokenizers后只需在`AutoTokenizer` API上打开 `use_fast=True`选项，即可调用C++实现的高性能分词算子，轻松获得超Python百余倍的文本处理加速，更多使用说明可参考[FastTokenizer文档](./fast_tokenizer)。
 
 #### ⚡️ FasterGeneration：高性能生成加速库
 

README_en.md

@@ -30,6 +30,8 @@
 
 ## News 📢
 
+* 🔥 **2022.11.12 PaddleNLP added AutoPrompt and won the first place in FewCLUE!**
+  * 🥇 The PaddleNLP team has open-sourced the **AutoPrompt** solution, which is based on the open-source Wenxin ERNIE pre-training language model, combined with domain pre-training and automated prompt learning technology, and ranked first in FewCLUE (a authoritative few-sample learning contest) with a model with 291M parameters. [see details](https://mp.weixin.qq.com/s/_JPiAzFA1f0BZ0igdv-EKA).
 * 🔥 **2022.10.27 [PaddleNLP v2.4.2](https://github.com/PaddlePaddle/PaddleNLP/releases/tag/v2.4.2) Released!**
   * NLG Upgrade: 📄 Release [**Solution of Text Summarization**](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/applications/text_summarization/pegasus) based on Pegasus;❓ Release [**Solution of Problem Generation**](./examples/question_generation), providing **general problem generation pre-trained model** based on Baidu's UNIMO Text and large-scale multi domain problem generation dataset。Supporting high-performance inference ability based on FasterGeneration , and covering the whole process of training , inference and deployment.
 * 🔥 **2022.10.14 [PaddleNLP v2.4.1](https://github.com/PaddlePaddle/PaddleNLP/releases/tag/v2.4.1) Released!**
@@ -221,17 +223,17 @@ For more details please refer to [Speech Command Analysis](./applications/speech
 
 ### High Performance Distributed Training and Inference
 
-#### ⚡ FasterTokenizer: High Performance Text Preprocessing Library
+#### ⚡ FastTokenizer: High Performance Text Preprocessing Library
 
 <div align="center">
     <img src="https://user-images.githubusercontent.com/11793384/168407921-b4395b1d-44bd-41a0-8c58-923ba2b703ef.png" width="400">
 </div>
 
 ```python
-AutoTokenizer.from_pretrained("ernie-3.0-medium-zh", use_faster=True)
+AutoTokenizer.from_pretrained("ernie-3.0-medium-zh", use_fast=True)
 ```
 
-Set `use_faster=True` to use C++ Tokenizer kernel to achieve 100x faster on text pre-processing. For more usage please refer to [FasterTokenizer](./faster_tokenizer).
+Set `use_fast=True` to use C++ Tokenizer kernel to achieve 100x faster on text pre-processing. For more usage please refer to [FastTokenizer](./fast_tokenizer).
 
 #### ⚡ FasterGeneration: High Perforance Generation Library
 

applications/neural_search/recall/in_batch_negative/README.md

@@ -229,6 +229,9 @@ python -u -m paddle.distributed.launch --gpus "0,1,2,3" \
 * `recall_num`: 对 1 个文本召回的相似文本数量
 * `similar_text_pair_file`: 由相似文本对构成的评估集
 * `corpus_file`: 召回库数据 corpus_file
+* `use_recompute`: 使用Recompute策略，用于节省显存，是一种以时间换空间的技术
+* `use_gradient_cache`: 使用Gradient Cache策略，用于节省显存，是一种以时间换空间的技术
+* `chunk_numbers`: 使用Gradient Cache策略的参数，表示的是同一个批次的样本分几次执行
 
 也可以使用bash脚本：
 

applications/neural_search/recall/in_batch_negative/batch_negative/model.py

@@ -60,14 +60,14 @@ class SemanticIndexBatchNeg(SemanticIndexBase):
                                    title_cls_embedding,
                                    transpose_y=True)
 
-        # substract margin from all positive samples cosine_sim()
+        # Substract margin from all positive samples cosine_sim()
         margin_diag = paddle.full(shape=[query_cls_embedding.shape[0]],
                                   fill_value=self.margin,
                                   dtype=paddle.get_default_dtype())
 
         cosine_sim = cosine_sim - paddle.diag(margin_diag)
 
-        # scale cosine to ease training converge
+        # Scale cosine to ease training converge
         cosine_sim *= self.sacle
 
         labels = paddle.arange(0, query_cls_embedding.shape[0], dtype='int64')
@@ -76,3 +76,56 @@ class SemanticIndexBatchNeg(SemanticIndexBase):
         loss = F.cross_entropy(input=cosine_sim, label=labels)
 
         return loss
+
+
+class SemanticIndexCacheNeg(SemanticIndexBase):
+
+    def __init__(self,
+                 pretrained_model,
+                 dropout=None,
+                 margin=0.3,
+                 scale=30,
+                 output_emb_size=None):
+        super().__init__(pretrained_model, dropout, output_emb_size)
+        self.margin = margin
+        # Used scaling cosine similarity to ease converge
+        self.sacle = scale
+
+    def forward(self,
+                query_input_ids,
+                title_input_ids,
+                query_token_type_ids=None,
+                query_position_ids=None,
+                query_attention_mask=None,
+                title_token_type_ids=None,
+                title_position_ids=None,
+                title_attention_mask=None):
+
+        query_cls_embedding = self.get_pooled_embedding(query_input_ids,
+                                                        query_token_type_ids,
+                                                        query_position_ids,
+                                                        query_attention_mask)
+
+        title_cls_embedding = self.get_pooled_embedding(title_input_ids,
+                                                        title_token_type_ids,
+                                                        title_position_ids,
+                                                        title_attention_mask)
+
+        cosine_sim = paddle.matmul(query_cls_embedding,
+                                   title_cls_embedding,
+                                   transpose_y=True)
+
+        # Substract margin from all positive samples cosine_sim()
+        margin_diag = paddle.full(shape=[query_cls_embedding.shape[0]],
+                                  fill_value=self.margin,
+                                  dtype=cosine_sim.dtype)
+
+        cosine_sim = cosine_sim - paddle.diag(margin_diag)
+
+        # Scale cosine to ease training converge
+        cosine_sim *= self.sacle
+
+        labels = paddle.arange(0, query_cls_embedding.shape[0], dtype='int64')
+        labels = paddle.reshape(labels, shape=[-1, 1])
+
+        return [cosine_sim, labels, query_cls_embedding, title_cls_embedding]

applications/neural_search/recall/in_batch_negative/scripts/run_build_index.sh

@@ -1,6 +1,20 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 # GPU version
 root_dir="checkpoints/inbatch" 
-python -u -m paddle.distributed.launch --gpus "3" --log_dir "recall_log/" \
+python -u -m paddle.distributed.launch --gpus "0" --log_dir "recall_log/" \
         recall.py \
         --device gpu \
         --recall_result_dir "recall_result_dir" \
@@ -11,7 +25,7 @@ python -u -m paddle.distributed.launch --gpus "3" --log_dir "recall_log/" \
         --hnsw_ef 100 \
         --batch_size 64 \
         --output_emb_size 256\
-        --max_seq_length 60 \
+        --max_seq_length 64 \
         --recall_num 50 \
         --similar_text_pair "recall/dev.csv" \
         --corpus_file "recall/corpus.csv" 

applications/neural_search/recall/in_batch_negative/train_batch_neg.py

@@ -17,14 +17,17 @@ import random
 import time
 import numpy as np
 import paddle
+import paddle.nn.functional as F
 from functools import partial
+
 from paddlenlp.utils.log import logger
 from paddlenlp.data import Tuple, Pad
 from paddlenlp.datasets import load_dataset, MapDataset
 from paddlenlp.transformers import AutoModel, AutoTokenizer
 from paddlenlp.transformers import LinearDecayWithWarmup
+
 from base_model import SemanticIndexBase
-from batch_negative.model import SemanticIndexBatchNeg
+from batch_negative.model import SemanticIndexBatchNeg, SemanticIndexCacheNeg
 from data import read_text_pair, convert_example, create_dataloader, gen_id2corpus, gen_text_file
 from ann_util import build_index
 
@@ -89,6 +92,15 @@ parser.add_argument("--evaluate_result", type=str, default='evaluate_result.txt'
                     help="evaluate_result")
 parser.add_argument('--evaluate', action='store_true',
                     help='whether evaluate while training')
+parser.add_argument("--use_amp", action="store_true", help="Whether to use AMP.")
+parser.add_argument("--amp_loss_scale", default=32768, type=float,help="The value of scale_loss for fp16. This is only used for AMP training.")
+parser.add_argument("--use_recompute",
+                        action='store_true',
+                        help="Using the recompute to scale up the batch size and save the memory.")
+parser.add_argument("--use_gradient_cache",
+                        action='store_true',
+                        help="Using the gradient cache to scale up the batch size and save the memory.")
+parser.add_argument("--chunk_numbers",type=int,default=50,help="The number of the chunks for model")
 args = parser.parse_args()
 # yapf: enable
 
@@ -161,6 +173,179 @@ def evaluate(model, corpus_data_loader, query_data_loader, recall_result_file,
     return float(recall_N[1])
 
 
+def train(train_data_loader, model, optimizer, lr_scheduler, rank,
+          corpus_data_loader, query_data_loader, recall_result_file, text_list,
+          id2corpus, tokenizer):
+    global_step = 0
+    best_recall = 0.0
+    tic_train = time.time()
+    for epoch in range(1, args.epochs + 1):
+        for step, batch in enumerate(train_data_loader, start=1):
+            query_input_ids, query_token_type_ids, title_input_ids, title_token_type_ids = batch
+
+            loss = model(query_input_ids=query_input_ids,
+                         title_input_ids=title_input_ids,
+                         query_token_type_ids=query_token_type_ids,
+                         title_token_type_ids=title_token_type_ids)
+
+            global_step += 1
+            if global_step % args.log_steps == 0 and rank == 0:
+                print(
+                    "global step %d, epoch: %d, batch: %d, loss: %.5f, speed: %.2f step/s"
+                    % (global_step, epoch, step, loss, args.log_steps /
+                       (time.time() - tic_train)))
+                tic_train = time.time()
+            loss.backward()
+            optimizer.step()
+            lr_scheduler.step()
+            optimizer.clear_grad()
+            if not args.evaluate:
+                if global_step % args.save_steps == 0 and rank == 0:
+                    save_dir = os.path.join(args.save_dir,
+                                            "model_%d" % global_step)
+                    if not os.path.exists(save_dir):
+                        os.makedirs(save_dir)
+                    save_param_path = os.path.join(save_dir,
+                                                   'model_state.pdparams')
+                    paddle.save(model.state_dict(), save_param_path)
+                    tokenizer.save_pretrained(save_dir)
+        if args.evaluate and rank == 0:
+            print("evaluating")
+            recall_5 = evaluate(model, corpus_data_loader, query_data_loader,
+                                recall_result_file, text_list, id2corpus)
+            if recall_5 > best_recall:
+                best_recall = recall_5
+
+                save_dir = os.path.join(args.save_dir, "model_best")
+                if not os.path.exists(save_dir):
+                    os.makedirs(save_dir)
+                save_param_path = os.path.join(save_dir, 'model_state.pdparams')
+                paddle.save(model.state_dict(), save_param_path)
+                tokenizer.save_pretrained(save_dir)
+                with open(os.path.join(save_dir, "train_result.txt"),
+                          'a',
+                          encoding='utf-8') as fp:
+                    fp.write('epoch=%d, global_step: %d, recall: %s\n' %
+                             (epoch, global_step, recall_5))
+
+
+def gradient_cache_train(train_data_loader, model, optimizer, lr_scheduler,
+                         rank, tokenizer):
+
+    if args.use_amp:
+        scaler = paddle.amp.GradScaler(init_loss_scaling=args.amp_loss_scale)
+
+    if args.batch_size % args.chunk_numbers == 0:
+        chunk_numbers = args.chunk_numbers
+    else:
+        raise Exception(
+            f" Batch_size {args.batch_size} must divides chunk_numbers {args.chunk_numbers} without producing a remainder "
+        )
+
+    def split(inputs, chunk_numbers, axis=0):
+        if inputs.shape[0] % chunk_numbers == 0:
+            return paddle.split(inputs, chunk_numbers, axis=0)
+        else:
+            return paddle.split(inputs, inputs.shape[0], axis=0)
+
+    global_step = 0
+    tic_train = time.time()
+    for epoch in range(1, args.epochs + 1):
+        for step, batch in enumerate(train_data_loader, start=1):
+            # Separate large batches into several sub batches
+            chunked_x = [split(t, chunk_numbers, axis=0) for t in batch]
+            sub_batchs = [list(s) for s in zip(*chunked_x)]
+
+            all_grads = []
+            all_CUDA_rnd_state = []
+            all_query = []
+            all_title = []
+
+            for sub_batch in sub_batchs:
+                all_reps = []
+                all_labels = []
+                sub_query_input_ids, sub_query_token_type_ids, sub_title_input_ids, sub_title_token_type_ids = sub_batch
+                with paddle.amp.auto_cast(
+                        args.use_amp,
+                        custom_white_list=["layer_norm", "softmax", "gelu"]):
+
+                    with paddle.no_grad():
+                        sub_CUDA_rnd_state = paddle.framework.random.get_cuda_rng_state(
+                        )
+                        all_CUDA_rnd_state.append(sub_CUDA_rnd_state)
+                        sub_cosine_sim, sub_label, query_embedding, title_embedding = model(
+                            query_input_ids=sub_query_input_ids,
+                            title_input_ids=sub_title_input_ids,
+                            query_token_type_ids=sub_query_token_type_ids,
+                            title_token_type_ids=sub_title_token_type_ids)
+                        all_reps.append(sub_cosine_sim)
+                        all_labels.append(sub_label)
+                        all_title.append(title_embedding)
+                        all_query.append(query_embedding)
+
+                model_reps = paddle.concat(all_reps, axis=0)
+                model_title = paddle.concat(all_title)
+                model_query = paddle.concat(all_query)
+
+                model_title = model_title.detach()
+                model_query = model_query.detach()
+
+                model_query.stop_gradient = False
+                model_title.stop_gradient = False
+                model_reps.stop_gradient = False
+
+                model_label = paddle.concat(all_labels, axis=0)
+                loss = F.cross_entropy(input=model_reps, label=model_label)
+                loss.backward()
+                # Store gradients
+                all_grads.append(model_reps.grad)
+
+            for sub_batch, CUDA_state, grad in zip(sub_batchs,
+                                                   all_CUDA_rnd_state,
+                                                   all_grads):
+
+                sub_query_input_ids, sub_query_token_type_ids, sub_title_input_ids, sub_title_token_type_ids = sub_batch
+                paddle.framework.random.set_cuda_rng_state(CUDA_state)
+                # Recompute the forward propogation
+                sub_cosine_sim, sub_label, query_embedding, title_embedding = model(
+                    query_input_ids=sub_query_input_ids,
+                    title_input_ids=sub_title_input_ids,
+                    query_token_type_ids=sub_query_token_type_ids,
+                    title_token_type_ids=sub_title_token_type_ids)
+                # Chain rule
+                surrogate = paddle.dot(sub_cosine_sim, grad)
+                # Backward propogation
+                if args.use_amp:
+                    scaled = scaler.scale(surrogate)
+                    scaled.backward()
+                else:
+                    surrogate.backward()
+            # Update model parameters
+            if args.use_amp:
+                scaler.minimize(optimizer, scaled)
+            else:
+                optimizer.step()
+
+            global_step += 1
+            if global_step % args.log_steps == 0 and rank == 0:
+                print(
+                    "global step %d, epoch: %d, batch: %d, loss: %.5f, speed: %.2f step/s"
+                    % (global_step, epoch, step, loss, args.log_steps /
+                       (time.time() - tic_train)))
+                tic_train = time.time()
+
+            lr_scheduler.step()
+            optimizer.clear_grad()
+
+            if global_step % args.save_steps == 0 and rank == 0:
+                save_dir = os.path.join(args.save_dir, "model_%d" % global_step)
+                if not os.path.exists(save_dir):
+                    os.makedirs(save_dir)
+                save_param_path = os.path.join(save_dir, 'model_state.pdparams')
+                paddle.save(model.state_dict(), save_param_path)
+                tokenizer.save_pretrained(save_dir)
+
+
 def do_train():
     paddle.set_device(args.device)
     rank = paddle.distributed.get_rank()
@@ -173,7 +358,8 @@ def do_train():
                             data_path=args.train_set_file,
                             lazy=False)
 
-    pretrained_model = AutoModel.from_pretrained(args.model_name_or_path)
+    pretrained_model = AutoModel.from_pretrained(
+        args.model_name_or_path, enable_recompute=args.use_recompute)
 
     tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
 
@@ -197,11 +383,16 @@ def do_train():
                                           batch_size=args.batch_size,
                                           batchify_fn=batchify_fn,
                                           trans_fn=trans_func)
-
-    model = SemanticIndexBatchNeg(pretrained_model,
-                                  margin=args.margin,
-                                  scale=args.scale,
-                                  output_emb_size=args.output_emb_size)
+    if (args.use_gradient_cache):
+        model = SemanticIndexCacheNeg(pretrained_model,
+                                      margin=args.margin,
+                                      scale=args.scale,
+                                      output_emb_size=args.output_emb_size)
+    else:
+        model = SemanticIndexBatchNeg(pretrained_model,
+                                      margin=args.margin,
+                                      scale=args.scale,
+                                      output_emb_size=args.output_emb_size)
 
     if args.init_from_ckpt and os.path.isfile(args.init_from_ckpt):
         state_dict = paddle.load(args.init_from_ckpt)
@@ -262,57 +453,13 @@ def do_train():
         weight_decay=args.weight_decay,
         apply_decay_param_fun=lambda x: x in decay_params)
 
-    global_step = 0
-    best_recall = 0.0
-    tic_train = time.time()
-    for epoch in range(1, args.epochs + 1):
-        for step, batch in enumerate(train_data_loader, start=1):
-            query_input_ids, query_token_type_ids, title_input_ids, title_token_type_ids = batch
-
-            loss = model(query_input_ids=query_input_ids,
-                         title_input_ids=title_input_ids,
-                         query_token_type_ids=query_token_type_ids,
-                         title_token_type_ids=title_token_type_ids)
-
-            global_step += 1
-            if global_step % args.log_steps == 0 and rank == 0:
-                print(
-                    "global step %d, epoch: %d, batch: %d, loss: %.5f, speed: %.2f step/s"
-                    % (global_step, epoch, step, loss, 10 /
-                       (time.time() - tic_train)))
-                tic_train = time.time()
-            loss.backward()
-            optimizer.step()
-            lr_scheduler.step()
-            optimizer.clear_grad()
-            if not args.evaluate:
-                if global_step % args.save_steps == 0 and rank == 0:
-                    save_dir = os.path.join(args.save_dir,
-                                            "model_%d" % global_step)
-                    if not os.path.exists(save_dir):
-                        os.makedirs(save_dir)
-                    save_param_path = os.path.join(save_dir,
-                                                   'model_state.pdparams')
-                    paddle.save(model.state_dict(), save_param_path)
-                    tokenizer.save_pretrained(save_dir)
-        if args.evaluate and rank == 0:
-            print("evaluating")
-            recall_5 = evaluate(model, corpus_data_loader, query_data_loader,
-                                recall_result_file, text_list, id2corpus)
-            if recall_5 > best_recall:
-                best_recall = recall_5
-
-                save_dir = os.path.join(args.save_dir, "model_best")
-                if not os.path.exists(save_dir):
-                    os.makedirs(save_dir)
-                save_param_path = os.path.join(save_dir, 'model_state.pdparams')
-                paddle.save(model.state_dict(), save_param_path)
-                tokenizer.save_pretrained(save_dir)
-                with open(os.path.join(save_dir, "train_result.txt"),
-                          'a',
-                          encoding='utf-8') as fp:
-                    fp.write('epoch=%d, global_step: %d, recall: %s\n' %
-                             (epoch, global_step, recall_5))
+    if (args.use_gradient_cache):
+        gradient_cache_train(train_data_loader, model, optimizer, lr_scheduler,
+                             rank, tokenizer)
+    else:
+        train(train_data_loader, model, optimizer, lr_scheduler, rank,
+              corpus_data_loader, query_data_loader, recall_result_file,
+              text_list, id2corpus, tokenizer)
 
 
 if __name__ == "__main__":

applications/text_classification/README.md

@@ -115,6 +115,7 @@
 
 【快速开始】
 - 快速开启多分类任务参见 👉 [语义索引-多分类指南](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/applications/text_classification/multi_class/retrieval_based#readme)
+- 快速开启多标签分类任务参见 👉 [语义索引-多标签分类指南](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/applications/text_classification/multi_label/retrieval_based#readme)
 - 快速开启层次分类任务参见 👉 [语义索引-层次分类指南](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/applications/text_classification/hierarchical/retrieval_based#readme)
 
 

applications/text_classification/doccano.py

@@ -115,14 +115,15 @@ def do_convert():
                     level_labels = label.split(args.separator)
                     for i in range(len(level_labels)):
                         l = args.separator.join(level_labels[:i + 1])
-                        label_dict.append(l)
+                        if l not in label_dict:
+                            label_dict.append(l)
                         if l not in label_list:
                             label_list.append(l)
                 if args.dirty:
                     text = ' '.join(text.strip().split('\t')[:-1])
                 else:
                     text = ' '.join(text.strip().split('\t'))
-                example = text + '\t' + ','.join(label_list) + '\n'
+                example = text + '\t' + ','.join(label_dict) + '\n'
             examples.append(example)
 
     if not args.dirty and not args.valid:

applications/text_classification/hierarchical/analysis/README.md

@@ -121,7 +121,7 @@ Text	Label	Prediction
 ### 单词级别可解释性分析
 本项目开源模型的词级别可解释性分析Notebook，提供LIME、Integrated Gradient、GradShap 三种分析方法，支持分析微调后模型的预测结果，开发者可以通过更改**数据目录**和**模型目录**在自己的任务中使用Jupyter Notebook进行数据分析。
 
-运行 [word_interpret.ipynb](./word_interpreter.ipynb) 代码，即可分析影响样本预测结果的关键词以及可视化所有词对预测结果的贡献情况，颜色越深代表这个词对预测结果影响越大：
+运行 [word_interpret.ipynb](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/applications/text_classification/hierarchical/analysis/README.md) 代码，即可分析影响样本预测结果的关键词以及可视化所有词对预测结果的贡献情况，颜色越深代表这个词对预测结果影响越大：
 <div align="center">
     <img src="https://user-images.githubusercontent.com/63761690/195334753-78cc2dc8-a5ba-4460-9fde-3b1bb704c053.png" width="1000">
 </div>

applications/text_classification/hierarchical/analysis/word_interpret.ipynb

@@ -112,6 +112,7 @@
     "            items = line.strip().split('\\t')\n",
     "            if items[0] == 'Text':\n",
     "                continue\n",
+    "            items[0] = items[0][:MAX_LENGTH-2]\n",
     "            if len(items) == 3:\n",
     "                yield {'text': items[0], 'label': items[1], 'predict': items[2]}\n",
     "            elif len(items) == 2:\n",

applications/text_classification/hierarchical/deploy/paddle_serving/README.md

@@ -54,10 +54,10 @@ pip install paddle-serving-server-gpu==0.8.3.post112 -i https://pypi.tuna.tsingh
 - 默认开启国内清华镜像源来加速下载，如果您使用 HTTP 代理可以关闭(-i https://pypi.tuna.tsinghua.edu.cn/simple)
 - 更多wheel包请参考[serving官网文档](https://github.com/PaddlePaddle/Serving/blob/develop/doc/Latest_Packages_CN.md)
 
-### 安装FasterTokenizer文本处理加速库（可选）
-推荐安装faster_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
+### 安装FastTokenizer文本处理加速库（可选）
+推荐安装fast_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
 ```shell
-pip install faster_tokenizer
+pip install fast_tokenizer
 ```
 
 
@@ -147,7 +147,7 @@ I0727 06:50:34.988327 43126 analysis_predictor.cc:1007] ======= optimize end ===
 I0727 06:50:34.992336 43126 naive_executor.cc:102] ---  skip [feed], feed -> token_type_ids
 I0727 06:50:34.992357 43126 naive_executor.cc:102] ---  skip [feed], feed -> input_ids
 I0727 06:50:34.993671 43126 naive_executor.cc:102] ---  skip [linear_75.tmp_1], fetch -> fetch
-[2022-07-27 06:50:35,954] [ WARNING] - Can't find the faster_tokenizers package, please ensure install faster_tokenizers correctly. You can install faster_tokenizers by `pip install faster_tokenizers`(Currently only work for linux platform).
+[2022-07-27 06:50:35,954] [ WARNING] - Can't find the fast_tokenizer package, please ensure install fast_tokenizer correctly. You can install fast_tokenizer by `pip install fast_tokenizer`.
 [2022-07-27 06:50:35,954] [    INFO] - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load 'ernie-3.0-medium-zh'.
 [2022-07-27 06:50:35,954] [    INFO] - Already cached /root/.paddlenlp/models/ernie-3.0-medium-zh/ernie_3.0_medium_zh_vocab.txt
 [OP Object] init success

applications/text_classification/hierarchical/deploy/paddle_serving/service.py

@@ -49,7 +49,7 @@ class Op(Op):
 
     def init_op(self):
         self.tokenizer = AutoTokenizer.from_pretrained(args.model_name,
-                                                       use_faster=True)
+                                                       use_fast=True)
         # Output nodes may differ from model to model
         # You can see the output node name in the conf.prototxt file of serving_server
         self.fetch_names = [

applications/text_classification/hierarchical/deploy/predictor/README.md

@@ -11,7 +11,7 @@
 
 如果基于GPU部署，请先确保机器已正确安装NVIDIA相关驱动和基础软件，确保CUDA >= 11.2，CuDNN >= 8.2，并使用以下命令安装所需依赖:
 ```shell
-python -m pip install onnxruntime-gpu onnx onnxconverter-common psutil
+python -m pip install onnxruntime-gpu onnx onnxconverter-common==1.9.0 psutil
 ```
 
 如果基于CPU部署，请使用如下命令安装所需依赖:
@@ -19,10 +19,10 @@ python -m pip install onnxruntime-gpu onnx onnxconverter-common psutil
 python -m pip install onnxruntime psutil
 ```
 
-安装FasterTokenizer文本处理加速库（可选）
-推荐安装faster_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
+安装FastTokenizer文本处理加速库（可选）
+推荐安装fast_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
 ```shell
-pip install faster_tokenizer
+pip install fast_tokenizer
 ```
 
 ## 基于GPU部署推理样例

applications/text_classification/hierarchical/deploy/predictor/predictor.py

@@ -123,7 +123,7 @@ class Predictor(object):
 
     def __init__(self, args, label_list):
         self.tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path,
-                                                       use_faster=True)
+                                                       use_fast=True)
         self.label_list = label_list
         self.batch_size = args.batch_size
         self.max_seq_length = args.max_seq_length

applications/text_classification/hierarchical/deploy/triton_serving/README.md

@@ -48,13 +48,13 @@ python3 -m pip install paddlepaddle-gpu paddlenlp -i https://mirror.baidu.com/py
 3. 更多关于PaddleNLP安装的详细教程请查看[Installation](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/docs/get_started/installation.rst)。
 
 
-### 安装FasterTokenizers文本处理加速库（可选）
+### 安装FastTokenizers文本处理加速库（可选）
 
-推荐安装faster_tokenizers可以得到更极致的文本处理效率，进一步提升服务性能。
+推荐安装fast_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
 
-在容器内安装 faster_tokenizers
+在容器内安装 fast_tokenizer
 ```shell
-python3 -m pip install faster_tokenizers
+python3 -m pip install fast_tokenizer
 ```
 
 
@@ -105,7 +105,7 @@ models
     └── config.pbtxt
 ```
 
-模型配置文件config.pbtxt配置细节请参见[Triton Server Model Configuration](https://github.com/triton-inference-server/server/blob/main/docs/model_configuration.md)
+模型配置文件config.pbtxt配置细节请参见[Triton Server Model Configuration](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md)
 
 ## 部署模型
 

applications/text_classification/hierarchical/deploy/triton_serving/models/tokenizer/1/model.py

@@ -48,7 +48,7 @@ class TritonPythonModel(object):
           * model_name: Model name
         """
         self.tokenizer = AutoTokenizer.from_pretrained("ernie-3.0-medium-zh",
-                                                       use_faster=True)
+                                                       use_fast=True)
         # You must parse model_config. JSON string is not parsed here
         self.model_config = json.loads(args['model_config'])
         print("model_config:", self.model_config)

applications/text_classification/hierarchical/few-shot/README.md

@@ -65,9 +65,9 @@
 
    内存: 630 GB
 
-3. PaddlePaddle 版本：2.3.1
+3. PaddlePaddle 版本：2.4rc
 
-4. PaddleNLP 版本：2.3.5 (develop)
+4. PaddleNLP 版本：2.4.3
 
 5. 评估设置
 
@@ -77,6 +77,7 @@
   - 微调
 
     ```
+    cd ../
     python train.py --dataset_dir "./data/" --save_dir "./checkpoints" --max_seq_length 128 --model_name "ernie-3.0-base-zh" --batch_size 8 --learning_rate 3e-5 --epochs 100 --logging_steps 5 --early_stop --early_stop_num 20
     ```
 
@@ -91,7 +92,7 @@
    | model_name | 训练方式 | Micro F1分数 | Macro F1分数 |
    | ---------- | ------- | ----------- | ----------- |
    | ernie-3.0-base-zh | 微调学习 | 0.7172 | 0.3821 |
-   | ernie-3.0-base-zh | 提示学习 | 0.8855 | 0.8443 |
+   | ernie-3.0-base-zh | 提示学习 | 0.8945 | 0.8516 |
 
 
 <a name="定制训练"></a>
@@ -102,10 +103,10 @@
 <a name="运行环境"></a>
 ### 3.1 运行环境
 
-- python >= 3.6
-- paddlepaddle > 2.3（2.4版本发布前推荐安装[develop版本](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/develop/install/pip/linux-pip.html)）
-- paddlenlp >= 2.3.5
-- paddle2onnx >= 1.0.0rc3
+- python >= 3.7
+- paddlepaddle >= 2.4rc
+- paddlenlp >= 2.4.3
+- paddle2onnx >= 1.0.3
 
 <a name="代码结构"></a>
 ### 3.2 代码结构
@@ -222,12 +223,12 @@ python train.py \
 --do_export \
 --num_train_epochs 100 \
 --logging_steps 5 \
+--save_total_limit 1 \
 --per_device_eval_batch_size 32 \
 --per_device_train_batch_size 8 \
 --metric_for_best_model macro_f1_score \
 --load_best_model_at_end \
---evaluation_strategy epoch \
---save_strategy epoch
+--eval_steps 100
 ```
 **多卡训练**
 
@@ -247,12 +248,12 @@ python -u -m paddle.distributed.launch --gpus 0,1,2,3 train.py \
 --do_export \
 --num_train_epochs 100 \
 --logging_steps 5 \
+--save_total_limit 1 \
 --per_device_eval_batch_size 32 \
 --per_device_train_batch_size 8 \
 --metric_for_best_model macro_f1_score \
 --load_best_model_at_end \
---evaluation_strategy epoch \
---save_strategy epoch
+--eval_steps 100
 ```
 
 可配置参数说明：
@@ -273,6 +274,7 @@ python -u -m paddle.distributed.launch --gpus 0,1,2,3 train.py \
 - `do_export`: 是否在运行结束时将模型导出为静态图，保存路径为`output_dir/export`。
 - `num_train_epochs`: 训练的最大轮数。
 - `max_steps`: 训练的最大步数。此设置将会覆盖`num_train_epochs`。
+- `save_total_limit`: 模型检查点保存数量。
 - `device`: 使用的设备，默认为`gpu`。
 - `eval_steps`: 评估模型的间隔步数。
 - `logging_steps`: 打印日志的间隔步数。
@@ -352,9 +354,9 @@ python infer.py --model_path_prefix checkpoints/export/model --data_dir ./data -
 可配置参数说明：
 
 - `model_path_prefix`: 导出的静态图模型路径及文件前缀。
-- `model_name_or_path`: 内置预训练模型名，或者模型参数配置目录路径，用于加载tokenizer。默认为`ernie-3.0-base-zh`。
+- `model_name`: 内置预训练模型名，用于加载tokenizer。默认为`ernie-3.0-base-zh`。
 - `data_dir`: 待推理数据所在路径，数据应存放在该目录下的`data.txt`文件。
-- `max_seq_length`: 最大句子长度，超过该长度的文本将被截断，不足的以Pad补全。提示文本不会被截断。
+- `max_length`: 最大句子长度，超过该长度的文本将被截断，不足的以Pad补全。提示文本不会被截断。
 - `batch_size`: 每次预测的样本数量。
 - `device`: 选择推理设备，包括`cpu`和`gpu`。默认为`gpu`。
 - `device_id`: 指定GPU设备ID。

applications/text_classification/hierarchical/few-shot/infer.py

@@ -14,23 +14,24 @@
 
 import os
 import six
+import json
 import psutil
 import argparse
 
 import numpy as np
 
 from paddlenlp.utils.log import logger
-from paddlenlp.prompt import AutoTemplate, Verbalizer, InputExample
-from paddlenlp.transformers import AutoTokenizer
+from paddlenlp.prompt import AutoTemplate, PromptDataCollatorWithPadding
+from paddlenlp.transformers import AutoTokenizer, AutoModelForMaskedLM
 import paddle2onnx
 import onnxruntime as ort
 
 # yapf: disable
 parser = argparse.ArgumentParser()
 parser.add_argument("--model_path_prefix", type=str, required=True, help="The path prefix of inference model to be used.")
-parser.add_argument("--model_name_or_path", default="ernie-3.0-base-zh", type=str, help="The directory or name of model.")
+parser.add_argument("--model_name", default="ernie-3.0-base-zh", type=str, help="The name of pretrained model.")
 parser.add_argument("--data_dir", default=None, type=str, help="The path to the prediction data, including label.txt and data.txt.")
-parser.add_argument("--max_seq_length", default=128, type=int, help="The maximum total input sequence length after tokenization.")
+parser.add_argument("--max_length", default=128, type=int, help="The maximum total input sequence length after tokenization.")
 parser.add_argument("--use_fp16", action='store_true', help="Whether to use fp16 inference, only takes effect when deploying on gpu.")
 parser.add_argument("--batch_size", default=200, type=int, help="Batch size per GPU/CPU for predicting.")
 parser.add_argument("--num_threads", default=psutil.cpu_count(logical=False), type=int, help="num_threads for cpu.")
@@ -103,12 +104,6 @@ class InferBackend(object):
                                                   'device_id':
                                                   device_id
                                               }])
-        self.input_handles = [
-            self.predictor.get_inputs()[0].name,
-            self.predictor.get_inputs()[1].name,
-            self.predictor.get_inputs()[2].name
-        ]
-
         if device == "gpu":
             try:
                 assert 'CUDAExecutionProvider' in self.predictor.get_providers()
@@ -122,27 +117,52 @@ class InferBackend(object):
         logger.info(">>> [InferBackend] Engine Created ...")
 
     def infer(self, input_dict: dict):
-        input_dict = {
-            k: v
-            for k, v in input_dict.items() if k in self.input_handles
-        }
         result = self.predictor.run(None, input_dict)
         return result
 
 
 class HierachicalPredictor(object):
 
-    def __init__(self, args, label_list):
-        self._label_list = label_list
-        self._tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
-        self._max_seq_length = args.max_seq_length
-        self._batch_size = args.batch_size
-        self.inference_backend = InferBackend(args.model_path_prefix,
-                                              args.device, args.device_id,
-                                              args.use_fp16, args.num_threads)
-        self._template = AutoTemplate.load_from(
-            os.path.dirname(args.model_path_prefix), self._tokenizer,
-            args.max_seq_length)
+    def __init__(self, args):
+        self.args = args
+        self.tokenizer = AutoTokenizer.from_pretrained(args.model_name)
+        self.model = AutoModelForMaskedLM.from_pretrained(args.model_name)
+        self.template, self.labels, self.input_handles = self.post_init()
+        self.collate_fn = PromptDataCollatorWithPadding(
+            self.tokenizer,
+            padding=True,
+            return_tensors="np",
+            return_attention_mask=True)
+
+        self.inference_backend = InferBackend(self.args.model_path_prefix,
+                                              self.args.device,
+                                              self.args.device_id,
+                                              self.args.use_fp16,
+                                              self.args.num_threads)
+
+    def post_init(self):
+        export_path = os.path.dirname(self.args.model_path_prefix)
+        template_path = os.path.join(export_path, "template_config.json")
+        with open(template_path, "r") as fp:
+            prompt = json.load(fp)
+            template = AutoTemplate.create_from(prompt, self.tokenizer,
+                                                self.args.max_length,
+                                                self.model)
+        keywords = template.extract_template_keywords(template.prompt)
+        inputs = [
+            "input_ids", "token_type_ids", "position_ids", "attention_mask",
+            "masked_positions"
+        ]
+        if "soft" in keywords:
+            inputs.append("soft_token_ids")
+        if "encoder" in keywords:
+            inputs.append("encoder_ids")
+        verbalizer_path = os.path.join(export_path, "verbalizer_config.json")
+        with open(verbalizer_path, "r") as fp:
+            label_words = json.load(fp)
+            labels = sorted(list(label_words.keys()))
+
+        return template, labels, inputs
 
     def predict(self, input_data: list):
         encoded_inputs = self.preprocess(input_data)
@@ -155,14 +175,27 @@ class HierachicalPredictor(object):
         infer_data = self.inference_backend.infer(input_dict)
         return infer_data
 
-    def infer_batch(self, encoded_inputs):
-        num_sample = len(encoded_inputs["input_ids"])
+    def infer_batch(self, inputs):
+        num_sample = len(inputs)
         infer_data = None
         num_infer_data = None
-        for idx in range(0, num_sample, self._batch_size):
-            l, r = idx, idx + self._batch_size
-            keys = encoded_inputs.keys()
-            input_dict = {k: encoded_inputs[k][l:r] for k in keys}
+        for index in range(0, num_sample, self.args.batch_size):
+            left, right = index, index + self.args.batch_size
+            batch_dict = self.collate_fn(inputs[left:right])
+            input_dict = {}
+            for key in self.input_handles:
+                value = batch_dict[key]
+                if key == "attention_mask":
+                    if value.ndim == 2:
+                        value = (1 - value[:, np.newaxis, np.newaxis, :]) * -1e4
+                    elif value.ndim != 4:
+                        raise ValueError(
+                            "Expect attention mask with ndim=2 or 4, but get ndim={}"
+                            .format(value.ndim))
+                    value = value.astype("float32")
+                else:
+                    value = value.astype("int64")
+                input_dict[key] = value
             results = self._infer(input_dict)
             if infer_data is None:
                 infer_data = [[x] for x in results]
@@ -175,16 +208,8 @@ class HierachicalPredictor(object):
         return infer_data
 
     def preprocess(self, input_data: list):
-        text = [InputExample(text_a=x) for x in input_data]
-        inputs = [self._template.wrap_one_example(x) for x in text]
-        inputs = {
-            "input_ids":
-            np.array([x["input_ids"] for x in inputs], dtype="int64"),
-            "mask_ids":
-            np.array([x["mask_ids"] for x in inputs], dtype="int64"),
-            "soft_token_ids":
-            np.array([x["soft_token_ids"] for x in inputs], dtype="int64")
-        }
+        text = [{"text_a": x} for x in input_data]
+        inputs = [self.template(x) for x in text]
         return inputs
 
     @staticmethod
@@ -197,7 +222,7 @@ class HierachicalPredictor(object):
         label_ids = np.argwhere(probs > threshold)
         labels = [[] for _ in range(probs.shape[0])]
         for idx, label_id in label_ids:
-            labels[idx].append(self._label_list[label_id])
+            labels[idx].append(self.labels[label_id])
         return {"label": labels}
 
     def printer(self, result, input_data):
@@ -212,12 +237,10 @@ if __name__ == "__main__":
     for arg_name, arg_value in vars(args).items():
         logger.info("{:20}: {}".format(arg_name, arg_value))
 
-    export_path = os.path.dirname(args.model_path_prefix)
-    labels, _ = Verbalizer.load_from(export_path)
+    predictor = HierachicalPredictor(args)
 
     text_dir = os.path.join(args.data_dir, "data.txt")
     with open(text_dir, "r", encoding="utf-8") as f:
         text_list = [x.strip() for x in f.readlines()]
 
-    predictor = HierachicalPredictor(args, labels)
     predictor.predict(text_list)

applications/text_classification/hierarchical/few-shot/requirements_cpu.txt

@@ -0,0 +1,5 @@
+psutil
+paddlepaddle>=2.4rc
+paddlenlp>=2.4.3
+paddle2onnx>=1.0.3
+onnxruntime

applications/text_classification/hierarchical/few-shot/requirements_gpu.txt

@@ -0,0 +1,7 @@
+psutil
+paddlepaddle-gpu>=2.4rc
+paddlenlp>=2.4.3
+paddle2onnx>=1.0.3
+onnxruntime-gpu
+onnx
+onnxconverter-common

applications/text_classification/hierarchical/few-shot/train.py

@@ -15,6 +15,7 @@
 from dataclasses import dataclass, field
 import os
 import sys
+from collections import defaultdict
 
 import paddle
 import paddle.nn.functional as F
@@ -41,9 +42,6 @@ from metric import MetricReport
 class DataArguments:
     data_dir: str = field(default="./data", metadata={"help": "The dataset dictionary includes train.txt, dev.txt, test.txt, label.txt and data.txt (optional) files."})
     prompt: str = field(default=None, metadata={"help": "The input prompt for tuning."})
-    soft_encoder: str = field(default="lstm", metadata={"help": "The encoder type of soft template, `lstm`, `mlp` or None."})
-    encoder_hidden_size: int = field(default=200, metadata={"help": "The dimension of soft embeddings."})
-
 
 @dataclass
 class ModelArguments:
@@ -67,17 +65,20 @@ def main():
     tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path)
 
     # Define the template for preprocess and the verbalizer for postprocess.
-    template = AutoTemplate.create_from(
-        data_args.prompt,
-        tokenizer,
-        training_args.max_seq_length,
-        model=model,
-        prompt_encoder=data_args.soft_encoder,
-        encoder_hidden_size=data_args.encoder_hidden_size)
-    logger.info("Using template: {}".format(template.template))
+    template = AutoTemplate.create_from(data_args.prompt,
+                                        tokenizer,
+                                        training_args.max_seq_length,
+                                        model=model)
+    logger.info("Using template: {}".format(template.prompt))
 
     label_file = os.path.join(data_args.data_dir, "label.txt")
-    verbalizer = SoftVerbalizer.from_file(tokenizer, model, label_file)
+    with open(label_file, "r", encoding="utf-8") as fp:
+        label_words = defaultdict(list)
+        for line in fp:
+            data = line.strip().split("==")
+            word = data[1] if len(data) > 1 else data[0].split("##")[-1]
+            label_words[data[0]].append(word)
+    verbalizer = SoftVerbalizer(label_words, tokenizer, model)
 
     # Load the few-shot datasets.
     train_ds, dev_ds, test_ds = load_local_dataset(
@@ -139,11 +140,24 @@ def main():
 
     # Export static model.
     if training_args.do_export:
+        template = prompt_model.template
+        template_keywords = template.extract_template_keywords(template.prompt)
         input_spec = [
-            InputSpec(shape=[None, None], dtype="int64"),  # input_ids
-            InputSpec(shape=[None, None], dtype="int64"),  # mask_ids
-            InputSpec(shape=[None, None], dtype="int64"),  # soft_token_ids
+            InputSpec(shape=[None, None], dtype="int64"),  # input_ids,
+            InputSpec(shape=[None, None], dtype="int64"),  # token_type_ids
+            InputSpec(shape=[None, None], dtype="int64"),  # position_ids
+            InputSpec(shape=[None, None, None, None],
+                      dtype="float32")  # attention_mask
         ]
+        if "mask" in template_keywords:
+            input_spec.append(InputSpec(shape=[None],
+                                        dtype="int64"))  # masked_positions
+        if "soft" in template_keywords:
+            input_spec.append(InputSpec(shape=[None, None],
+                                        dtype="int64"))  # soft_token_ids
+        if "encoder" in template_keywords:
+            input_spec.append(InputSpec(shape=[None, None],
+                                        dtype="int64"))  # encoder_ids
         export_path = os.path.join(training_args.output_dir, 'export')
         trainer.export_model(export_path,
                              input_spec=input_spec,

applications/text_classification/hierarchical/few-shot/utils.py

@@ -15,7 +15,6 @@
 import os
 
 from paddlenlp.datasets import load_dataset
-from paddlenlp.prompt import InputExample
 
 
 def load_local_dataset(data_path, splits, label_list):
@@ -39,14 +38,14 @@ def load_local_dataset(data_path, splits, label_list):
             for idx, line in enumerate(fp):
                 data = line.strip().split("\t")
                 if len(data) == 1:
-                    yield InputExample(text_a=data[0])
+                    yield {"text_a": data[0]}
                 else:
                     text, label = data
                     label = label.strip().split(",")
                     label = [
                         float(1) if x in label else float(0) for x in label_list
                     ]
-                    yield InputExample(text_a=text, labels=label)
+                    yield {"text_a": text, "labels": label}
 
     split_map = {"train": "train.txt", "dev": "dev.txt", "test": "test.txt"}
     datasets = []

applications/text_classification/multi_class/README.md

@@ -188,67 +188,93 @@ data.txt(待预测数据文件)，需要预测标签的文本数据。
 #### 2.4.1 预训练模型微调
 
 
-使用CPU/GPU训练，默认为GPU训练，使用CPU训练只需将设备参数配置改为`--device "cpu"`：
+使用CPU/GPU训练，默认为GPU训练，使用CPU训练只需将设备参数配置改为`--device cpu`：
 ```shell
 python train.py \
-    --dataset_dir "data" \
-    --device "gpu" \
+    --model_name_or_path ernie-3.0-medium-zh \
+    --data_dir ./data/ \
+    --output_dir checkpoint \
+    --device gpu \
+    --learning_rate 3e-5 \
+    --num_train_epochs 100 \
+    --early_stopping_patience 4 \
     --max_seq_length 128 \
-    --model_name "ernie-3.0-medium-zh" \
-    --batch_size 32 \
-    --early_stop \
-    --epochs 100
+    --per_device_eval_batch_size 32 \
+    --per_device_train_batch_size 32 \
+    --num_train_epochs 100 \
+    --do_train \
+    --do_eval \
+    --metric_for_best_model accuracy \
+    --load_best_model_at_end \
+    --evaluation_strategy epoch \
+    --save_strategy epoch \
+    --save_total_limit 1
 ```
 
 如果在CPU环境下训练，可以指定`nproc_per_node`参数进行多核训练：
 ```shell
-python -m paddle.distributed.launch --nproc_per_node 8 --backend "gloo" train.py \
-    --dataset_dir "data" \
-    --device "cpu" \
+python -m paddle.distributed.launch --nproc_per_node 8 --backend gloo train.py \
+    --model_name_or_path ernie-3.0-medium-zh \
+    --data_dir ./data/ \
+    --output_dir checkpoint \
+    --device cpu \
+    --learning_rate 3e-5 \
+    --num_train_epochs 100 \
     --max_seq_length 128 \
-    --model_name "ernie-3.0-medium-zh" \
-    --batch_size 32 \
-    --early_stop \
-    --epochs 100
+    --per_device_eval_batch_size 32 \
+    --per_device_train_batch_size 32 \
+    --num_train_epochs 100 \
+    --early_stopping_patience 4 \
+    --do_train \
+    --do_eval \
+    --metric_for_best_model accuracy \
+    --load_best_model_at_end \
+    --evaluation_strategy epoch \
+    --save_strategy epoch \
+    --save_total_limit 1
 ```
 
-如果在GPU环境中使用，可以指定`gpus`参数进行单卡/多卡训练。使用多卡训练可以指定多个GPU卡号，例如 --gpus "0,1"。如果设备只有一个GPU卡号默认为0，可使用`nvidia-smi`命令查看GPU使用情况:
+如果在GPU环境中使用，可以指定`gpus`参数进行单卡/多卡训练。使用多卡训练可以指定多个GPU卡号，例如 --gpus 0,1。如果设备只有一个GPU卡号默认为0，可使用`nvidia-smi`命令查看GPU使用情况:
 
 ```shell
 unset CUDA_VISIBLE_DEVICES
-python -m paddle.distributed.launch --gpus "0" train.py \
-    --dataset_dir "data" \
-    --device "gpu" \
+python -m paddle.distributed.launch --gpus 0,1 train.py \
+    --data_dir ./data/ \
+    --output_dir checkpoint \
+    --device cpu \
+    --learning_rate 3e-5 \
+    --num_train_epochs 100 \
     --max_seq_length 128 \
-    --model_name "ernie-3.0-medium-zh" \
-    --batch_size 32 \
-    --early_stop \
-    --epochs 100
+    --per_device_eval_batch_size 32 \
+    --per_device_train_batch_size 32 \
+    --num_train_epochs 100 \
+    --early_stopping_patience 4 \
+    --do_train \
+    --do_eval \
+    --metric_for_best_model accuracy \
+    --load_best_model_at_end \
+    --evaluation_strategy epoch \
+    --save_strategy epoch \
+    --save_total_limit 1
 ```
 
-可支持配置的参数：
+主要的配置的参数为：
+- `model_name_or_path`: 内置模型名，或者模型参数配置目录路径。默认为`ernie-3.0-base-zh`。
+- `data_dir`: 训练数据集路径，数据格式要求详见[数据标注](#数据标注)。
+- `output_dir`: 模型参数、训练日志和静态图导出的保存目录。
+- `max_seq_length`: 最大句子长度，超过该长度的文本将被截断，不足的以Pad补全。提示文本不会被截断。
+- `num_train_epochs`: 训练轮次，使用早停法时可以选择100
+- `early_stopping_patience`: 在设定的早停训练轮次内，模型在开发集上表现不再上升，训练终止；默认为4。
+- `learning_rate`: 预训练语言模型参数基础学习率大小，将与learning rate scheduler产生的值相乘作为当前学习率。
+- `do_train`: 是否进行训练。
+- `do_eval`: 是否进行评估。
+- `device`: 使用的设备，默认为`gpu`。
+- `per_device_train_batch_size`: 每次训练每张卡上的样本数量。可根据实际GPU显存适当调小/调大此配置。
+- `per_device_eval_batch_size`: 每次评估每张卡上的样本数量。可根据实际GPU显存适当调小/调大此配置。
 
-* `device`: 选用什么设备进行训练，选择cpu、gpu、xpu、npu。如使用gpu训练，可使用参数--gpus指定GPU卡号；默认为"gpu"。
-* `dataset_dir`：必须，本地数据集路径，数据集路径中应包含train.txt，dev.txt和label.txt文件;默认为None。
-* `save_dir`：保存训练模型的目录；默认保存在当前目录checkpoint文件夹下。
-* `max_seq_length`：分词器tokenizer使用的最大序列长度，ERNIE模型最大不能超过2048。请根据文本长度选择，通常推荐128、256或512，若出现显存不足，请适当调低这一参数；默认为128。
-* `model_name`：选择预训练模型,可选"ernie-1.0-large-zh-cw","ernie-3.0-xbase-zh", "ernie-3.0-base-zh", "ernie-3.0-medium-zh", "ernie-3.0-micro-zh", "ernie-3.0-mini-zh", "ernie-3.0-nano-zh", "ernie-2.0-base-en", "ernie-2.0-large-en","ernie-m-base","ernie-m-large"；默认为"ernie-3.0-medium-zh"。
-* `batch_size`：批处理大小，请结合显存情况进行调整，若出现显存不足，请适当调低这一参数；默认为32。
-* `learning_rate`：训练最大学习率；默认为3e-5。
-* `epochs`: 训练轮次，使用早停法时可以选择100；默认为10。
-* `early_stop`：选择是否使用早停法(EarlyStopping)，模型在开发集经过一定epoch后精度表现不再上升，训练终止；默认为False。
-* `early_stop_nums`：在设定的早停训练轮次内，模型在开发集上表现不再上升，训练终止；默认为4。
-* `logging_steps`: 训练过程中日志打印的间隔steps数，默认5。
-* `weight_decay`：控制正则项力度的参数，用于防止过拟合，默认为0.0。
-* `warmup`：是否使用学习率warmup策略，使用时应设置适当的训练轮次（epochs）；默认为False。
-* `warmup_steps`：学习率warmup策略的比例数，如果设为1000，则学习率会在1000steps数从0慢慢增长到learning_rate, 而后再缓慢衰减；默认为0。
-* `init_from_ckpt`: 模型初始checkpoint参数地址，默认None。
-* `seed`：随机种子，默认为3。
-* `train_file`：本地数据集中训练集文件名；默认为"train.txt"。
-* `dev_file`：本地数据集中开发集文件名；默认为"dev.txt"。
-* `label_file`：本地数据集中标签集文件名；默认为"label.txt"。
+训练脚本支持所有`TraingArguments`的参数，更多参数介绍可参考[TrainingArguments 参数介绍](https://paddlenlp.readthedocs.io/zh/latest/trainer.html#trainingarguments)。
 
-程序运行时将会自动进行训练，评估。同时训练过程中会自动保存开发集上最佳模型在指定的 `save_dir` 中，保存模型文件结构如下所示：
+程序运行时将会自动进行训练，评估。同时训练过程中会自动保存开发集上最佳模型在指定的 `output_dir` 中，保存模型文件结构如下所示：
 
 ```text
 checkpoint/
@@ -260,8 +286,8 @@ checkpoint/
 
 **NOTE:**
 
-* 如需恢复模型训练，则可以设置 `init_from_ckpt` ， 如 `init_from_ckpt=checkpoint/model_state.pdparams` 。
-* 如需训练英文文本分类任务，只需更换预训练模型参数 `model_name` 。英文训练任务推荐使用"ernie-2.0-base-en"、"ernie-2.0-large-en"。
+* 如需恢复模型训练，则可以设置 `resume_from_checkpoint` ， 如 `resume_from_checkpoint=./checkpoints/checkpoint-217` 。
+* 如需训练英文文本分类任务，只需更换预训练模型参数 `model_name_or_path` 。英文训练任务推荐使用"ernie-2.0-base-en"、"ernie-2.0-large-en"。
 * 英文和中文以外语言的文本分类任务，推荐使用基于96种语言（涵盖法语、日语、韩语、德语、西班牙语等几乎所有常见语言）进行预训练的多语言预训练模型"ernie-m-base"、"ernie-m-large"，详情请参见[ERNIE-M论文](https://arxiv.org/pdf/2012.15674.pdf)。
 
 #### 2.4.2 训练评估与模型优化

applications/text_classification/multi_class/analysis/README.md

@@ -119,7 +119,7 @@ Text	Label	Prediction
 ### 单词级别可解释性分析
 本项目开源模型的词级别可解释性分析Notebook，提供LIME、Integrated Gradient、GradShap 三种分析方法，支持分析微调后模型的预测结果，开发者可以通过更改**数据目录**和**模型目录**在自己的任务中使用Jupyter Notebook进行数据分析。
 
-运行 [word_interpret.ipynb](./word_interpreter.ipynb) 代码，即可分析影响样本预测结果的关键词以及可视化所有词对预测结果的贡献情况，颜色越深代表这个词对预测结果影响越大：
+运行 [word_interpret.ipynb](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/applications/text_classification/multi_class/analysis/README.md) 代码，即可分析影响样本预测结果的关键词以及可视化所有词对预测结果的贡献情况，颜色越深代表这个词对预测结果影响越大：
 <div align="center">
     <img src="https://user-images.githubusercontent.com/63761690/192739675-63145d59-23c6-416f-bf71-998fd4995254.png" width="1000">
 </div>

applications/text_classification/multi_class/analysis/evaluate.py

@@ -156,16 +156,17 @@ def evaluate():
     logger.info("Dev dataset size: {}".format(len(dev_ds)))
     logger.info("Accuracy in dev dataset: {:.2f}%".format(report['accuracy'] *
                                                           100))
-    logger.info("Top-2 accuracy in dev dataset: {:.2f}%".format(
-        top_k_accuracy_score(y_true=labels,
-                             y_score=probs,
-                             k=2,
-                             labels=list(range(len(label_list)))) * 100))
-    logger.info("Top-3 accuracy in dev dataset: {:.2f}%".format(
-        top_k_accuracy_score(y_true=labels,
-                             y_score=probs,
-                             k=3,
-                             labels=list(range(len(label_list)))) * 100))
+    if len(labels) > 2:
+        logger.info("Top-2 accuracy in dev dataset: {:.2f}%".format(
+            top_k_accuracy_score(y_true=labels,
+                                 y_score=probs,
+                                 k=2,
+                                 labels=list(range(len(label_list)))) * 100))
+        logger.info("Top-3 accuracy in dev dataset: {:.2f}%".format(
+            top_k_accuracy_score(y_true=labels,
+                                 y_score=probs,
+                                 k=3,
+                                 labels=list(range(len(label_list)))) * 100))
 
     for i, l in enumerate(label_list):
         logger.info("Class name: {}".format(l))

applications/text_classification/multi_class/analysis/word_interpret.ipynb

@@ -112,6 +112,7 @@
     "            items = line.strip().split('\\t')\n",
     "            if items[0] == 'Text':\n",
     "                continue\n",
+    "            items[0] = items[0][:MAX_LENGTH-2]\n",
     "            if len(items) == 3:\n",
     "                yield {'text': items[0], 'label': items[1], 'predict': items[2]}\n",
     "            elif len(items) == 2:\n",

applications/text_classification/multi_class/deploy/paddle_serving/README.md

@@ -53,10 +53,10 @@ pip install paddle-serving-server-gpu==0.8.3.post112 -i https://pypi.tuna.tsingh
 - 更多wheel包请参考[serving官网文档](https://github.com/PaddlePaddle/Serving/blob/develop/doc/Latest_Packages_CN.md)
 
 
-### 安装FasterTokenizer文本处理加速库（可选）
-推荐安装faster_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
+### 安装FastTokenizer文本处理加速库（可选）
+推荐安装fast_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
 ```shell
-pip install faster_tokenizer
+pip install fast_tokenizer
 ```
 
 
@@ -143,7 +143,7 @@ I0628 09:12:30.739985 74305 analysis_predictor.cc:1007] ======= optimize end ===
 I0628 09:12:30.776288 74305 naive_executor.cc:102] ---  skip [feed], feed -> token_type_ids
 I0628 09:12:30.779004 74305 naive_executor.cc:102] ---  skip [feed], feed -> input_ids
 I0628 09:12:30.787542 74305 naive_executor.cc:102] ---  skip [linear_147.tmp_1], fetch -> fetch
-[2022-06-28 09:12:32,879] [ WARNING] - Can't find the faster_tokenizers package, please ensure install faster_tokenizers correctly. You can install faster_tokenizers by `pip install faster_tokenizers`(Currently only work for linux platform).
+[2022-06-28 09:12:32,879] [ WARNING] - Can't find the fast_tokenizer package, please ensure install fast_tokenizer correctly. You can install fast_tokenizer by `pip install fast_tokenizer`.
 [2022-06-28 09:12:32,880] [    INFO] - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load 'ernie-3.0-medium-zh'.
 [2022-06-28 09:12:32,880] [    INFO] - Already cached /root/.paddlenlp/models/ernie-3.0-medium-zh/ernie_3.0_base_zh_vocab.txt
 [OP Object] init success

applications/text_classification/multi_class/deploy/paddle_serving/service.py

@@ -49,7 +49,7 @@ class Op(Op):
 
     def init_op(self):
         self.tokenizer = AutoTokenizer.from_pretrained(args.model_name,
-                                                       use_faster=True)
+                                                       use_fast=True)
         # Output nodes may differ from model to model
         # You can see the output node name in the conf.prototxt file of serving_server
         self.fetch_names = [

applications/text_classification/multi_class/deploy/predictor/README.md

@@ -12,7 +12,7 @@
 
 如果基于GPU部署，请先确保机器已正确安装NVIDIA相关驱动和基础软件，确保CUDA >= 11.2，CuDNN >= 8.2，并使用以下命令安装所需依赖:
 ```
-python -m pip install onnxruntime-gpu onnx onnxconverter-common
+python -m pip install onnxruntime-gpu onnx onnxconverter-common==1.9.0
 ```
 
 如果基于CPU部署，请使用如下命令安装所需依赖:
@@ -20,10 +20,10 @@ python -m pip install onnxruntime-gpu onnx onnxconverter-common
 python -m pip install onnxruntime
 ```
 
-安装FasterTokenizer文本处理加速库（可选）
-推荐安装faster_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
+安装FastTokenizer文本处理加速库（可选）
+推荐安装fast_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
 ```shell
-pip install faster_tokenizer
+pip install fast_tokenizer
 ```
 ## 基于GPU部署推理样例
 

applications/text_classification/multi_class/deploy/predictor/predictor.py

@@ -117,7 +117,7 @@ class Predictor(object):
 
     def __init__(self, args, label_list):
         self.tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path,
-                                                       use_faster=True)
+                                                       use_fast=True)
         self.label_list = label_list
         self.batch_size = args.batch_size
         self.max_seq_length = args.max_seq_length

applications/text_classification/multi_class/deploy/triton_serving/README.md

@@ -48,13 +48,13 @@ python3 -m pip install paddlepaddle-gpu paddlenlp -i https://mirror.baidu.com/py
 3. 更多关于PaddleNLP安装的详细教程请查看[Installation](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/docs/get_started/installation.rst)。
 
 
-### 安装FasterTokenizers文本处理加速库（可选）
+### 安装FastTokenizer文本处理加速库（可选）
 
-部署环境是Linux，推荐安装faster_tokenizers可以得到更极致的文本处理效率，进一步提升服务性能。
+部署环境是Linux，推荐安装fast_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
 
-在容器内安装 faster_tokenizers
+在容器内安装 fast_tokenizer
 ```shell
-python3 -m pip install faster_tokenizers
+python3 -m pip install fast_tokenizer
 ```
 
 
@@ -102,7 +102,7 @@ models
     └── config.pbtxt
 ```
 
-模型配置文件config.pbtxt配置细节请参见[Triton Server Model Configuration](https://github.com/triton-inference-server/server/blob/main/docs/model_configuration.md)
+模型配置文件config.pbtxt配置细节请参见[Triton Server Model Configuration](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md)
 
 ## 部署模型
 

applications/text_classification/multi_class/deploy/triton_serving/models/tokenizer/1/model.py

@@ -48,7 +48,7 @@ class TritonPythonModel(object):
           * model_name: Model name
         """
         self.tokenizer = AutoTokenizer.from_pretrained("ernie-3.0-medium-zh",
-                                                       use_faster=True)
+                                                       use_fast=True)
         # You must parse model_config. JSON string is not parsed here
         self.model_config = json.loads(args['model_config'])
         print("model_config:", self.model_config)

applications/text_classification/multi_class/few-shot/README.md

@@ -65,9 +65,9 @@
 
    内存: 630 GB
 
-3. PaddlePaddle 版本：2.3.1
+3. PaddlePaddle 版本：2.4rc
 
-4. PaddleNLP 版本：2.3.5 (develop)
+4. PaddleNLP 版本：2.4.3
 
 5. 评估设置
 
@@ -76,13 +76,14 @@
 - 微调
 
 ```
+cd ../
 python train.py --dataset_dir "./data/" --save_dir "./checkpoints" --max_seq_length 128 --model_name "ernie-3.0-base-zh" --batch_size 8 --learning_rate 3e-5 --epochs 100 --logging_steps 5 --early_stop
 ```
 
 - 提示学习
 
 ```
-python train.py --data_dir ./data/ --output_dir ./checkpoints/ --prompt "这条新闻写的是" --model_name_or_path ernie-3.0-base-zh --max_seq_length 128  --learning_rate 3e-5 --ppt_learning_rate 3e-4 --do_train --do_eval --num_train_epochs 100 --logging_steps 5 --per_device_eval_batch_size 32 --per_device_train_batch_size 8 --do_predict --metric_for_best_model accuracy --load_best_model_at_end --evaluation_strategy epoch --save_strategy epoch
+python train.py --data_dir ./data/ --output_dir ./checkpoints/ --prompt "这条新闻写的是" --model_name_or_path ernie-3.0-base-zh --max_seq_length 128  --learning_rate 3e-5 --ppt_learning_rate 3e-4 --do_train --do_eval --num_train_epochs 100 --logging_steps 5 --per_device_eval_batch_size 32 --per_device_train_batch_size 8 --do_predict --metric_for_best_model accuracy --load_best_model_at_end --evaluation_strategy epoch --save_strategy epoch --save_total_limit 1
 ```
 
 6. 精度评价指标：Accuracy
@@ -102,10 +103,10 @@ python train.py --data_dir ./data/ --output_dir ./checkpoints/ --prompt "这条
 <a name="运行环境"></a>
 ### 3.1 运行环境
 
-- python >= 3.6
-- paddlepaddle > 2.3 （2.4版本发布前推荐安装[develop版本](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/develop/install/pip/linux-pip.html)）
-- paddlenlp >= 2.3.5
-- paddle2onnx >= 1.0.0rc3
+- python >= 3.7
+- paddlepaddle >= 2.4rc
+- paddlenlp >= 2.4.3
+- paddle2onnx >= 1.0.3
 
 <a name="代码结构"></a>
 ### 3.2 代码结构
@@ -204,14 +205,15 @@ python train.py \
 --output_dir ./checkpoints/ \
 --prompt "这条新闻标题的主题是" \
 --max_seq_length 128  \
---learning_rate 3e-5 \
---ppt_learning_rate 3e-4 \
+--learning_rate 3e-6 \
+--ppt_learning_rate 3e-5 \
 --do_train \
 --do_eval \
 --use_rdrop \
 --max_steps 1000 \
 --eval_steps 10 \
 --logging_steps 5 \
+--save_total_limit 1 \
 --load_best_model_at_end True \
 --per_device_eval_batch_size 32 \
 --per_device_train_batch_size 8 \
@@ -227,8 +229,8 @@ python -u -m paddle.distributed.launch --gpus 0,1,2,3 train.py \
 --output_dir ./checkpoints/ \
 --prompt "这条新闻标题的主题是" \
 --max_seq_length 128  \
---learning_rate 3e-5 \
---ppt_learning_rate 3e-4 \
+--learning_rate 3e-6 \
+--ppt_learning_rate 3e-5 \
 --do_train \
 --do_eval \
 --use_rdrop \
@@ -236,6 +238,7 @@ python -u -m paddle.distributed.launch --gpus 0,1,2,3 train.py \
 --max_steps 1000 \
 --eval_steps 10 \
 --logging_steps 5 \
+--save_total_limit 1 \
 --load_best_model_at_end True \
 --per_device_eval_batch_size 32 \
 --per_device_train_batch_size 8 \
@@ -260,6 +263,7 @@ python -u -m paddle.distributed.launch --gpus 0,1,2,3 train.py \
 - `do_predict`: 是否进行预测。
 - `do_export`: 是否在运行结束时将模型导出为静态图，保存路径为`output_dir/export`。
 - `max_steps`: 训练的最大步数。此设置将会覆盖`num_train_epochs`。
+- `save_total_limit`: 模型检查点保存数量。
 - `eval_steps`: 评估模型的间隔步数。
 - `device`: 使用的设备，默认为`gpu`。
 - `logging_steps`: 打印日志的间隔步数。
@@ -335,9 +339,9 @@ python infer.py --model_path_prefix checkpoints/export/model --data_dir ./data -
 可配置参数说明：
 
 - `model_path_prefix`: 导出的静态图模型路径及文件前缀。
-- `model_name_or_path`: 内置预训练模型名，或者模型参数配置目录路径，用于加载tokenizer。默认为`ernie-3.0-base-zh`。
+- `model_name`: 内置预训练模型名，用于加载tokenizer。默认为`ernie-3.0-base-zh`。
 - `data_dir`: 待推理数据所在路径，数据应存放在该目录下的`data.txt`文件。
-- `max_seq_length`: 最大句子长度，超过该长度的文本将被截断，不足的以Pad补全。提示文本不会被截断。
+- `max_length`: 最大句子长度，超过该长度的文本将被截断，不足的以Pad补全。提示文本不会被截断。
 - `batch_size`: 每次预测的样本数量。
 - `device`: 选择推理设备，包括`cpu`和`gpu`。默认为`gpu`。
 - `device_id`: 指定GPU设备ID。

applications/text_classification/multi_class/few-shot/infer.py

@@ -14,23 +14,24 @@
 
 import os
 import six
+import json
 import psutil
 import argparse
 
 import numpy as np
 
 from paddlenlp.utils.log import logger
-from paddlenlp.prompt import AutoTemplate, Verbalizer, InputExample
-from paddlenlp.transformers import AutoTokenizer
+from paddlenlp.prompt import AutoTemplate, PromptDataCollatorWithPadding
+from paddlenlp.transformers import AutoTokenizer, AutoModelForMaskedLM
 import paddle2onnx
 import onnxruntime as ort
 
 # yapf: disable
 parser = argparse.ArgumentParser()
 parser.add_argument("--model_path_prefix", type=str, required=True, help="The path prefix of inference model to be used.")
-parser.add_argument("--model_name_or_path", default="ernie-3.0-base-zh", type=str, help="The directory or name of model.")
+parser.add_argument("--model_name", default="ernie-3.0-base-zh", type=str, help="The name of pretrained model.")
 parser.add_argument("--data_dir", default=None, type=str, help="The path to the prediction data, including label.txt and data.txt.")
-parser.add_argument("--max_seq_length", default=128, type=int, help="The maximum total input sequence length after tokenization.")
+parser.add_argument("--max_length", default=128, type=int, help="The maximum total input sequence length after tokenization.")
 parser.add_argument("--use_fp16", action='store_true', help="Whether to use fp16 inference, only takes effect when deploying on gpu.")
 parser.add_argument("--batch_size", default=200, type=int, help="Batch size per GPU/CPU for predicting.")
 parser.add_argument("--num_threads", default=psutil.cpu_count(logical=False), type=int, help="num_threads for cpu.")
@@ -103,11 +104,6 @@ class InferBackend(object):
                                                   'device_id':
                                                   device_id
                                               }])
-        self.input_handles = [
-            self.predictor.get_inputs()[0].name,
-            self.predictor.get_inputs()[1].name,
-            self.predictor.get_inputs()[2].name
-        ]
 
         if device == "gpu":
             try:
@@ -122,27 +118,53 @@ class InferBackend(object):
         logger.info(">>> [InferBackend] Engine Created ...")
 
     def infer(self, input_dict: dict):
-        input_dict = {
-            k: v
-            for k, v in input_dict.items() if k in self.input_handles
-        }
         result = self.predictor.run(None, input_dict)
         return result
 
 
 class MultiClassPredictor(object):
 
-    def __init__(self, args, label_list):
-        self._label_list = label_list
-        self._tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
-        self._max_seq_length = args.max_seq_length
-        self._batch_size = args.batch_size
-        self.inference_backend = InferBackend(args.model_path_prefix,
-                                              args.device, args.device_id,
-                                              args.use_fp16, args.num_threads)
-        self._template = AutoTemplate.load_from(
-            os.path.dirname(args.model_path_prefix), self._tokenizer,
-            args.max_seq_length)
+    def __init__(self, args):
+        self.args = args
+        self.tokenizer = AutoTokenizer.from_pretrained(args.model_name)
+        self.model = AutoModelForMaskedLM.from_pretrained(args.model_name)
+        self.template, self.labels, self.input_handles = self.post_init()
+        self.collate_fn = PromptDataCollatorWithPadding(
+            self.tokenizer,
+            padding=True,
+            return_tensors="np",
+            return_attention_mask=True)
+
+        self.inference_backend = InferBackend(self.args.model_path_prefix,
+                                              self.args.device,
+                                              self.args.device_id,
+                                              self.args.use_fp16,
+                                              self.args.num_threads)
+
+    def post_init(self):
+        export_path = os.path.dirname(self.args.model_path_prefix)
+        template_path = os.path.join(export_path, "template_config.json")
+        with open(template_path, "r") as fp:
+            prompt = json.load(fp)
+            template = AutoTemplate.create_from(prompt, self.tokenizer,
+                                                self.args.max_length,
+                                                self.model)
+        keywords = template.extract_template_keywords(template.prompt)
+        inputs = [
+            "input_ids", "token_type_ids", "position_ids", "attention_mask"
+        ]
+        if "mask" in keywords:
+            inputs.append("masked_positions")
+        if "soft" in keywords:
+            inputs.append("soft_token_ids")
+        if "encoder" in keywords:
+            inputs.append("encoder_ids")
+        verbalizer_path = os.path.join(export_path, "verbalizer_config.json")
+        with open(verbalizer_path, "r") as fp:
+            label_words = json.load(fp)
+            labels = sorted(list(label_words.keys()))
+
+        return template, labels, inputs
 
     def predict(self, input_data: list):
         encoded_inputs = self.preprocess(input_data)
@@ -155,14 +177,27 @@ class MultiClassPredictor(object):
         infer_data = self.inference_backend.infer(input_dict)
         return infer_data
 
-    def infer_batch(self, encoded_inputs):
-        num_sample = len(encoded_inputs["input_ids"])
+    def infer_batch(self, inputs):
+        num_sample = len(inputs)
         infer_data = None
         num_infer_data = None
-        for idx in range(0, num_sample, self._batch_size):
-            l, r = idx, idx + self._batch_size
-            keys = encoded_inputs.keys()
-            input_dict = {k: encoded_inputs[k][l:r] for k in keys}
+        for index in range(0, num_sample, self.args.batch_size):
+            left, right = index, index + self.args.batch_size
+            batch_dict = self.collate_fn(inputs[left:right])
+            input_dict = {}
+            for key in self.input_handles:
+                value = batch_dict[key]
+                if key == "attention_mask":
+                    if value.ndim == 2:
+                        value = (1 - value[:, np.newaxis, np.newaxis, :]) * -1e4
+                    elif value.ndim != 4:
+                        raise ValueError(
+                            "Expect attention mask with ndim=2 or 4, but get ndim={}"
+                            .format(value.ndim))
+                    value = value.astype("float32")
+                else:
+                    value = value.astype("int64")
+                input_dict[key] = value
             results = self._infer(input_dict)
             if infer_data is None:
                 infer_data = [[x] for x in results]
@@ -175,21 +210,13 @@ class MultiClassPredictor(object):
         return infer_data
 
     def preprocess(self, input_data: list):
-        text = [InputExample(text_a=x) for x in input_data]
-        inputs = [self._template.wrap_one_example(x) for x in text]
-        inputs = {
-            "input_ids":
-            np.array([x["input_ids"] for x in inputs], dtype="int64"),
-            "mask_ids":
-            np.array([x["mask_ids"] for x in inputs], dtype="int64"),
-            "soft_token_ids":
-            np.array([x["soft_token_ids"] for x in inputs], dtype="int64")
-        }
+        text = [{"text_a": x} for x in input_data]
+        inputs = [self.template(x) for x in text]
         return inputs
 
     def postprocess(self, infer_data):
         preds = np.argmax(infer_data[0], axis=-1)
-        labels = [self._label_list[x] for x in preds]
+        labels = [self.labels[x] for x in preds]
         return {"label": labels}
 
     def printer(self, result, input_data):
@@ -204,12 +231,10 @@ if __name__ == "__main__":
     for arg_name, arg_value in vars(args).items():
         logger.info("{:20}: {}".format(arg_name, arg_value))
 
-    export_path = os.path.dirname(args.model_path_prefix)
-    labels, _ = Verbalizer.load_from(export_path)
+    predictor = MultiClassPredictor(args)
 
     text_dir = os.path.join(args.data_dir, "data.txt")
     with open(text_dir, "r", encoding="utf-8") as f:
         text_list = [x.strip() for x in f.readlines()]
 
-    predictor = MultiClassPredictor(args, labels)
     predictor.predict(text_list)

applications/text_classification/multi_class/few-shot/requirements_cpu.txt

@@ -0,0 +1,5 @@
+psutil
+paddlepaddle>=2.4rc
+paddlenlp>=2.4.3
+paddle2onnx>=1.0.3
+onnxruntime

applications/text_classification/multi_class/few-shot/requirements_gpu.txt

@@ -0,0 +1,7 @@
+psutil
+paddlepaddle-gpu>=2.4rc
+paddlenlp>=2.4.3
+paddle2onnx>=1.0.3
+onnxruntime-gpu
+onnx
+onnxconverter-common

applications/text_classification/multi_class/few-shot/train.py

@@ -14,6 +14,9 @@
 
 from dataclasses import dataclass, field
 import os
+from collections import defaultdict
+
+import numpy as np
 
 import paddle
 from paddle.static import InputSpec
@@ -37,8 +40,6 @@ from utils import load_local_dataset
 class DataArguments:
     data_dir: str = field(default="./data/", metadata={"help": "Path to a dataset which includes train.txt, dev.txt, test.txt, label.txt and data.txt (optional)."})
     prompt: str = field(default=None, metadata={"help": "The input prompt for tuning."})
-    soft_encoder: str = field(default="lstm", metadata={"help": "The encoder type of soft template, `lstm`, `mlp` or None."})
-    encoder_hidden_size: int = field(default=200, metadata={"help": "The dimension of soft embeddings."})
 
 
 @dataclass
@@ -63,17 +64,20 @@ def main():
     tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path)
 
     # Define the template for preprocess and the verbalizer for postprocess.
-    template = AutoTemplate.create_from(
-        data_args.prompt,
-        tokenizer,
-        training_args.max_seq_length,
-        model=model,
-        prompt_encoder=data_args.soft_encoder,
-        encoder_hidden_size=data_args.encoder_hidden_size)
-    logger.info("Using template: {}".format(template.template))
+    template = AutoTemplate.create_from(data_args.prompt,
+                                        tokenizer,
+                                        training_args.max_seq_length,
+                                        model=model)
+    logger.info("Using template: {}".format(template.prompt))
 
     label_file = os.path.join(data_args.data_dir, "label.txt")
-    verbalizer = SoftVerbalizer.from_file(tokenizer, model, label_file)
+    with open(label_file, "r", encoding="utf-8") as fp:
+        label_words = defaultdict(list)
+        for line in fp:
+            data = line.strip().split("==")
+            word = data[1] if len(data) > 1 else data[0].split("##")[-1]
+            label_words[data[0]].append(word)
+    verbalizer = SoftVerbalizer(label_words, tokenizer, model)
 
     # Load the few-shot datasets.
     train_ds, dev_ds, test_ds = load_local_dataset(
@@ -133,11 +137,24 @@ def main():
 
     # Export static model.
     if training_args.do_export:
+        template = prompt_model.template
+        template_keywords = template.extract_template_keywords(template.prompt)
         input_spec = [
-            InputSpec(shape=[None, None], dtype="int64"),  # input_ids
-            InputSpec(shape=[None, None], dtype="int64"),  # mask_ids
-            InputSpec(shape=[None, None], dtype="int64"),  # soft_token_ids
+            InputSpec(shape=[None, None], dtype="int64"),  # input_ids,
+            InputSpec(shape=[None, None], dtype="int64"),  # token_type_ids
+            InputSpec(shape=[None, None], dtype="int64"),  # position_ids
+            InputSpec(shape=[None, None, None, None],
+                      dtype="float32")  # attention_mask
         ]
+        if "mask" in template_keywords:
+            input_spec.append(InputSpec(shape=[None],
+                                        dtype="int64"))  # masked_positions
+        if "soft" in template_keywords:
+            input_spec.append(InputSpec(shape=[None, None],
+                                        dtype="int64"))  # soft_token_ids
+        if "encoder" in template_keywords:
+            input_spec.append(InputSpec(shape=[None, None],
+                                        dtype="int64"))  # encoder_ids
         export_path = os.path.join(training_args.output_dir, 'export')
         trainer.export_model(export_path,
                              input_spec=input_spec,

applications/text_classification/multi_class/few-shot/utils.py

@@ -15,7 +15,6 @@
 import os
 
 from paddlenlp.datasets import load_dataset
-from paddlenlp.prompt import InputExample
 
 
 def load_local_dataset(data_path, splits, label_list):
@@ -38,10 +37,10 @@ def load_local_dataset(data_path, splits, label_list):
             for idx, line in enumerate(fp):
                 data = line.strip().split("\t")
                 if len(data) == 1:
-                    yield InputExample(text_a=data[0])
+                    yield {"text_a": data[0]}
                 else:
                     text, label = data
-                    yield InputExample(text_a=text, labels=label_list[label])
+                    yield {"text_a": text, "labels": label_list[label]}
 
     assert isinstance(splits, list) and len(splits) > 0
 

applications/text_classification/multi_class/train.py

@@ -12,206 +12,128 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import re
-import json
 import functools
-import random
-import time
 import os
-import argparse
-
-import numpy as np
+from dataclasses import dataclass, field
 
 import paddle
-import paddle.nn.functional as F
 from paddle.metric import Accuracy
-from paddle.io import DataLoader, BatchSampler, DistributedBatchSampler
 from paddlenlp.data import DataCollatorWithPadding
 from paddlenlp.datasets import load_dataset
-from paddlenlp.transformers import AutoModelForSequenceClassification, AutoTokenizer, LinearDecayWithWarmup
-from paddlenlp.utils.log import logger
+from paddlenlp.trainer import PdArgumentParser, EarlyStoppingCallback, TrainingArguments, Trainer
+from paddlenlp.transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+from utils import preprocess_function, read_local_dataset
+
+SUPPORTED_MODELS = [
+    "ernie-1.0-large-zh-cw", "ernie-3.0-xbase-zh", "ernie-3.0-base-zh",
+    "ernie-3.0-medium-zh", "ernie-3.0-micro-zh", "ernie-3.0-mini-zh",
+    "ernie-3.0-nano-zh", "ernie-2.0-base-en", "ernie-2.0-large-en",
+    "ernie-m-base", "ernie-m-large"
+]
 
-from utils import evaluate, preprocess_function, read_local_dataset
 
 # yapf: disable
-parser = argparse.ArgumentParser()
-parser.add_argument('--device', default="gpu", help="Select which device to train model, defaults to gpu.")
-parser.add_argument("--dataset_dir", required=True, default=None, type=str, help="Local dataset directory should include train.txt, dev.txt and label.txt")
-parser.add_argument("--save_dir", default="./checkpoint", type=str, help="The output directory where the model checkpoints will be written.")
-parser.add_argument("--max_seq_length", default=128, type=int, help="The maximum total input sequence length after tokenization. Sequences longer than this will be truncated, sequences shorter will be padded.")
-parser.add_argument('--model_name', default="ernie-3.0-medium-zh", help="Select model to train, defaults to ernie-3.0-medium-zh.",
-                    choices=["ernie-1.0-large-zh-cw","ernie-3.0-xbase-zh", "ernie-3.0-base-zh", "ernie-3.0-medium-zh", "ernie-3.0-micro-zh", "ernie-3.0-mini-zh", "ernie-3.0-nano-zh", "ernie-2.0-base-en", "ernie-2.0-large-en","ernie-m-base","ernie-m-large"])
-parser.add_argument("--batch_size", default=32, type=int, help="Batch size per GPU/CPU for training.")
-parser.add_argument("--learning_rate", default=3e-5, type=float, help="The initial learning rate for Adam.")
-parser.add_argument("--epochs", default=10, type=int, help="Total number of training epochs to perform.")
-parser.add_argument('--early_stop', action='store_true', help='Epoch before early stop.')
-parser.add_argument('--early_stop_nums', type=int, default=3, help='Number of epoch before early stop.')
-parser.add_argument("--logging_steps", default=5, type=int, help="The interval steps to logging.")
-parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
-parser.add_argument('--warmup', action='store_true', help="whether use warmup strategy")
-parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup steps over the training process.")
-parser.add_argument("--init_from_ckpt", type=str, default=None, help="The path of checkpoint to be loaded.")
-parser.add_argument("--seed", type=int, default=3, help="random seed for initialization")
-parser.add_argument("--train_file", type=str, default="train.txt", help="Train dataset file name")
-parser.add_argument("--dev_file", type=str, default="dev.txt", help="Dev dataset file name")
-parser.add_argument("--label_file", type=str, default="label.txt", help="Label file name")
-args = parser.parse_args()
+@dataclass
+class DataArguments:
+    data_dir: str = field(default="./data/", metadata={"help": "Path to a dataset which includes train.txt, dev.txt, test.txt, label.txt and data.txt (optional)."})
+    max_seq_length: int = field(default=128, metadata={"help": "Maximum number of tokens for the model"})
+    early_stopping_patience: int = field(default=4, metadata={"help": "Stop training when the specified metric worsens for early_stopping_patience evaluation calls"})
+
+@dataclass
+class ModelArguments:
+    model_name_or_path: str = field(default="ernie-3.0-base-zh", metadata={"help": "Build-in pretrained model name or the path to local model."})
 # yapf: enable
 
 
-def set_seed(seed):
-    """
-    Sets random seed
-    """
-    random.seed(seed)
-    np.random.seed(seed)
-    paddle.seed(seed)
-    os.environ['PYTHONHASHSEED'] = str(seed)
-
-
-def args_saving():
-    argsDict = args.__dict__
-    with open(os.path.join(args.save_dir, 'setting.txt'), 'w') as f:
-        f.writelines('------------------ start ------------------' + '\n')
-        for eachArg, value in argsDict.items():
-            f.writelines(eachArg + ' : ' + str(value) + '\n')
-        f.writelines('------------------- end -------------------')
-
-
-def train():
+def main():
     """
     Training a binary or multi classification model
     """
 
-    if not os.path.exists(args.save_dir):
-        os.makedirs(args.save_dir)
-    args_saving()
-    set_seed(args.seed)
-    paddle.set_device(args.device)
+    parser = PdArgumentParser(
+        (ModelArguments, DataArguments, TrainingArguments))
+    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
+    training_args.print_config(model_args, "Model")
+    training_args.print_config(data_args, "Data")
+    print(not os.path.isdir(model_args.model_name_or_path))
+    print(model_args.model_name_or_path not in SUPPORTED_MODELS)
+    if not os.path.isdir(
+            model_args.model_name_or_path
+    ) and model_args.model_name_or_path not in SUPPORTED_MODELS:
+        raise ValueError(
+            f"{model_args.model_name_or_path} is not a supported model type. Either use a local model path or select a model from {SUPPORTED_MODELS}"
+        )
 
-    rank = paddle.distributed.get_rank()
-    if paddle.distributed.get_world_size() > 1:
-        paddle.distributed.init_parallel_env()
+    paddle.set_device(training_args.device)
 
     # load and preprocess dataset
-
     label_list = {}
-    with open(os.path.join(args.dataset_dir, args.label_file),
+    with open(os.path.join(data_args.data_dir, "label.txt"),
               'r',
               encoding='utf-8') as f:
         for i, line in enumerate(f):
             l = line.strip()
             label_list[l] = i
+
     train_ds = load_dataset(read_local_dataset,
-                            path=os.path.join(args.dataset_dir,
-                                              args.train_file),
+                            path=os.path.join(data_args.data_dir, "train.txt"),
                             label_list=label_list,
                             lazy=False)
     dev_ds = load_dataset(read_local_dataset,
-                          path=os.path.join(args.dataset_dir, args.dev_file),
+                          path=os.path.join(data_args.data_dir, "dev.txt"),
                           label_list=label_list,
                           lazy=False)
 
-    tokenizer = AutoTokenizer.from_pretrained(args.model_name)
+    tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path)
     trans_func = functools.partial(preprocess_function,
                                    tokenizer=tokenizer,
-                                   max_seq_length=args.max_seq_length)
+                                   max_seq_length=data_args.max_seq_length)
     train_ds = train_ds.map(trans_func)
     dev_ds = dev_ds.map(trans_func)
-    # batchify dataset
-    collate_fn = DataCollatorWithPadding(tokenizer)
-    if paddle.distributed.get_world_size() > 1:
-        train_batch_sampler = DistributedBatchSampler(
-            train_ds, batch_size=args.batch_size, shuffle=True)
-    else:
-        train_batch_sampler = BatchSampler(train_ds,
-                                           batch_size=args.batch_size,
-                                           shuffle=True)
-    dev_batch_sampler = BatchSampler(dev_ds,
-                                     batch_size=args.batch_size,
-                                     shuffle=False)
-    train_data_loader = DataLoader(dataset=train_ds,
-                                   batch_sampler=train_batch_sampler,
-                                   collate_fn=collate_fn)
-    dev_data_loader = DataLoader(dataset=dev_ds,
-                                 batch_sampler=dev_batch_sampler,
-                                 collate_fn=collate_fn)
 
-    # define model
+    # Define model
     model = AutoModelForSequenceClassification.from_pretrained(
-        args.model_name, num_classes=len(label_list))
-    if args.init_from_ckpt and os.path.isfile(args.init_from_ckpt):
-        state_dict = paddle.load(args.init_from_ckpt)
-        model.set_dict(state_dict)
-    model = paddle.DataParallel(model)
+        model_args.model_name_or_path, num_classes=len(label_list))
 
-    num_training_steps = len(train_data_loader) * args.epochs
-    lr_scheduler = LinearDecayWithWarmup(args.learning_rate, num_training_steps,
-                                         args.warmup_steps)
+    # Define the metric function.
+    def compute_metrics(eval_preds):
+        metric = Accuracy()
+        correct = metric.compute(paddle.to_tensor(eval_preds.predictions),
+                                 paddle.to_tensor(eval_preds.label_ids))
+        metric.update(correct)
+        acc = metric.accumulate()
+        return {'accuracy': acc}
 
-    # Generate parameter names needed to perform weight decay.
-    # All bias and LayerNorm parameters are excluded.
-    decay_params = [
-        p.name for n, p in model.named_parameters()
-        if not any(nd in n for nd in ["bias", "norm"])
+    # Define the early-stopping callback.
+    callbacks = [
+        EarlyStoppingCallback(
+            early_stopping_patience=data_args.early_stopping_patience,
+            early_stopping_threshold=0.)
     ]
-    optimizer = paddle.optimizer.AdamW(
-        learning_rate=lr_scheduler,
-        parameters=model.parameters(),
-        weight_decay=args.weight_decay,
-        apply_decay_param_fun=lambda x: x in decay_params)
 
+    # Define loss function
     criterion = paddle.nn.loss.CrossEntropyLoss()
-    metric = Accuracy()
 
-    global_step = 0
-    best_acc = 0
-    early_stop_count = 0
-    tic_train = time.time()
+    # Define Trainer
+    trainer = Trainer(model=model,
+                      tokenizer=tokenizer,
+                      args=training_args,
+                      criterion=criterion,
+                      train_dataset=train_ds,
+                      eval_dataset=dev_ds,
+                      callbacks=callbacks,
+                      data_collator=DataCollatorWithPadding(tokenizer),
+                      compute_metrics=compute_metrics)
 
-    for epoch in range(1, args.epochs + 1):
-
-        if args.early_stop and early_stop_count >= args.early_stop_nums:
-            logger.info("Early stop!")
-            break
-
-        for step, batch in enumerate(train_data_loader, start=1):
-            labels = batch.pop("labels")
-            logits = model(**batch)
-            loss = criterion(logits, labels)
-
-            loss.backward()
-            optimizer.step()
-            if args.warmup:
-                lr_scheduler.step()
-            optimizer.clear_grad()
-
-            global_step += 1
-            if global_step % args.logging_steps == 0 and rank == 0:
-                logger.info(
-                    "global step %d, epoch: %d, batch: %d, loss: %.5f, speed: %.2f step/s"
-                    % (global_step, epoch, step, loss, args.logging_steps /
-                       (time.time() - tic_train)))
-                tic_train = time.time()
-
-        early_stop_count += 1
-        acc = evaluate(model, criterion, metric, dev_data_loader)
-        save_best_path = args.save_dir
-        if not os.path.exists(save_best_path):
-            os.makedirs(save_best_path)
-
-        # save models
-        if acc > best_acc:
-            early_stop_count = 0
-            best_acc = acc
-            model._layers.save_pretrained(save_best_path)
-            tokenizer.save_pretrained(save_best_path)
-        logger.info("Current best accuracy: %.5f" % (best_acc))
-
-    logger.info("Final best accuracy: %.5f" % (best_acc))
-    logger.info("Save best accuracy text classification model in %s" %
-                (args.save_dir))
+    if training_args.do_train:
+        train_result = trainer.train(resume_from_checkpoint=None)
+        metrics = train_result.metrics
+        trainer.save_model()
+        trainer.log_metrics("train", metrics)
+        trainer.save_metrics("train", metrics)
+        trainer.save_state()
 
 
 if __name__ == "__main__":
-    train()
+    main()

applications/text_classification/multi_class/utils.py

@@ -14,40 +14,6 @@
 
 import numpy as np
 
-import paddle
-import paddle.nn.functional as F
-from paddlenlp.utils.log import logger
-
-
-@paddle.no_grad()
-def evaluate(model, criterion, metric, data_loader):
-    """
-    Given a dataset, it evaluates model and computes the metric.
-    Args:
-        model(obj:`paddle.nn.Layer`): A model to classify texts.
-        criterion(obj:`paddle.nn.Layer`): It can compute the loss.
-        metric(obj:`paddle.metric.Metric`): The evaluation metric.
-        data_loader(obj:`paddle.io.DataLoader`): The dataset loader which generates batches.
-    """
-
-    model.eval()
-    metric.reset()
-    losses = []
-    for batch in data_loader:
-        labels = batch.pop("labels")
-        logits = model(**batch)
-        loss = criterion(logits, labels)
-        losses.append(loss.numpy())
-        correct = metric.compute(logits, labels)
-        metric.update(correct)
-
-    acc = metric.accumulate()
-    logger.info("eval loss: %.5f, acc: %.5f" % (np.mean(losses), acc))
-    model.train()
-    metric.reset()
-
-    return acc
-
 
 def preprocess_function(examples, tokenizer, max_seq_length, is_test=False):
     """

applications/text_classification/multi_label/README.md

@@ -36,7 +36,7 @@
 对于大多数多标签分类任务，我们推荐使用预训练模型微调作为首选的文本分类方案，多标签分类项目中还提供 提示学习(小样本)和语义索引的两种全流程文本分类方案满足不同开发者需求，更多技术细节请参见[文本分类技术特色介绍](../README.md)。
 
 - 【标注成本高、标注样本较少的小样本场景】 👉 [提示学习多标签分类方案](./few-shot#readme)
-
+- 【标签类别不固定场景、标签类别众多】 👉 [语义索引多分类方案](./retrieval_based#readme)
 <a name="快速开始"></a>
 
 ## 2. 快速开始

applications/text_classification/multi_label/analysis/README.md

@@ -116,7 +116,7 @@ Text	Label	Prediction
 ### 单词级别可解释性分析
 本项目开源模型的词级别可解释性分析Notebook，提供LIME、Integrated Gradient、GradShap 三种分析方法，支持分析微调后模型的预测结果，开发者可以通过更改**数据目录**和**模型目录**在自己的任务中使用Jupyter Notebook进行数据分析。
 
-运行 [word_interpret.ipynb](./word_interpreter.ipynb) 代码，即可分析影响样本预测结果的关键词以及可视化所有词对预测结果的贡献情况，颜色越深代表这个词对预测结果影响越大：
+运行 [word_interpret.ipynb](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/applications/text_classification/multi_label/analysis/README.md) 代码，即可分析影响样本预测结果的关键词以及可视化所有词对预测结果的贡献情况，颜色越深代表这个词对预测结果影响越大：
 <div align="center">
     <img src="https://user-images.githubusercontent.com/63761690/192739675-63145d59-23c6-416f-bf71-998fd4995254.png" width="1000">
 </div>

applications/text_classification/multi_label/deploy/paddle_serving/README.md

@@ -51,10 +51,10 @@ pip install paddle-serving-server-gpu==0.8.3.post112 -i https://pypi.tuna.tsingh
 - 默认开启国内清华镜像源来加速下载，如果您使用 HTTP 代理可以关闭(-i https://pypi.tuna.tsinghua.edu.cn/simple)
 - 更多wheel包请参考[serving官网文档](https://github.com/PaddlePaddle/Serving/blob/develop/doc/Latest_Packages_CN.md)
 
-### 安装FasterTokenizer文本处理加速库（可选）
-推荐安装faster_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
+### 安装FastTokenizer文本处理加速库（可选）
+推荐安装fast_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
 ```shell
-pip install faster_tokenizer
+pip install fast_tokenizer
 ```
 
 
@@ -142,7 +142,7 @@ I0625 16:44:36.563802 40218 analysis_predictor.cc:1007] ======= optimize end ===
 I0625 16:44:36.571702 40218 naive_executor.cc:102] ---  skip [feed], feed -> token_type_ids
 I0625 16:44:36.571728 40218 naive_executor.cc:102] ---  skip [feed], feed -> input_ids
 I0625 16:44:36.574352 40218 naive_executor.cc:102] ---  skip [linear_147.tmp_1], fetch -> fetch
-[2022-06-25 16:44:37,545] [ WARNING] - Can't find the faster_tokenizers package, please ensure install faster_tokenizers correctly. You can install faster_tokenizers by `pip install faster_tokenizers`(Currently only work for linux platform).
+[2022-06-25 16:44:37,545] [ WARNING] - Can't find the fast_tokenizer package, please ensure install fast_tokenizer correctly. You can install fast_tokenizer by `pip install fast_tokenizer`.
 [2022-06-25 16:44:37,546] [    INFO] - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load 'ernie-3.0-medium-zh'.
 [2022-06-25 16:44:37,546] [    INFO] - Already cached /root/.paddlenlp/models/ernie-3.0-medium-zh/ernie_3.0_base_zh_vocab.txt
 [OP Object] init success

applications/text_classification/multi_label/deploy/paddle_serving/service.py

@@ -49,7 +49,7 @@ class Op(Op):
 
     def init_op(self):
         self.tokenizer = AutoTokenizer.from_pretrained(args.model_name,
-                                                       use_faster=True)
+                                                       use_fast=True)
         # Output nodes may differ from model to model
         # You can see the output node name in the conf.prototxt file of serving_server
         self.fetch_names = [

applications/text_classification/multi_label/deploy/predictor/README.md

@@ -12,7 +12,7 @@
 
 如果基于GPU部署，请先确保机器已正确安装NVIDIA相关驱动和基础软件，确保CUDA >= 11.2，CuDNN >= 8.2，并使用以下命令安装所需依赖:
 ```shell
-python -m pip install onnxruntime-gpu onnx onnxconverter-common
+python -m pip install onnxruntime-gpu onnx onnxconverter-common==1.9.0
 ```
 
 如果基于CPU部署，请使用如下命令安装所需依赖:
@@ -20,10 +20,10 @@ python -m pip install onnxruntime-gpu onnx onnxconverter-common
 python -m pip install onnxruntime
 ```
 
-安装FasterTokenizer文本处理加速库（可选）
-推荐安装faster_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
+安装FastTokenizer文本处理加速库（可选）
+推荐安装fast_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
 ```shell
-pip install faster_tokenizer
+pip install fast_tokenizer
 ```
 ## 基于GPU部署推理样例
 

applications/text_classification/multi_label/deploy/predictor/predictor.py

@@ -124,7 +124,7 @@ class Predictor(object):
 
     def __init__(self, args, label_list):
         self.tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path,
-                                                       use_faster=True)
+                                                       use_fast=True)
         self.label_list = label_list
         self.batch_size = args.batch_size
         self.max_seq_length = args.max_seq_length

applications/text_classification/multi_label/deploy/triton_serving/README.md

@@ -48,13 +48,13 @@ python3 -m pip install paddlepaddle-gpu paddlenlp -i https://mirror.baidu.com/py
 3. 更多关于PaddleNLP安装的详细教程请查看[Installation](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/docs/get_started/installation.rst)。
 
 
-### 安装FasterTokenizers文本处理加速库（可选）
+### 安装FastTokenizer文本处理加速库（可选）
 
-推荐安装faster_tokenizers可以得到更极致的文本处理效率，进一步提升服务性能。
+推荐安装fast_tokenizer可以得到更极致的文本处理效率，进一步提升服务性能。
 
-在容器内安装 faster_tokenizers
+在容器内安装 fast_tokenizer
 ```shell
-python3 -m pip install faster_tokenizers
+python3 -m pip install fast_tokenizer
 ```
 
 
@@ -104,7 +104,7 @@ models
     └── config.pbtxt
 ```
 
-模型配置文件config.pbtxt配置细节请参见[Triton Server Model Configuration](https://github.com/triton-inference-server/server/blob/main/docs/model_configuration.md)
+模型配置文件config.pbtxt配置细节请参见[Triton Server Model Configuration](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md)
 
 ## 部署模型
 

applications/text_classification/multi_label/deploy/triton_serving/models/tokenizer/1/model.py

@@ -48,7 +48,7 @@ class TritonPythonModel(object):
           * model_name: Model name
         """
         self.tokenizer = AutoTokenizer.from_pretrained("ernie-3.0-medium-zh",
-                                                       use_faster=True)
+                                                       use_fast=True)
         # You must parse model_config. JSON string is not parsed here
         self.model_config = json.loads(args['model_config'])
         print("model_config:", self.model_config)

applications/text_classification/multi_label/few-shot/README.md

@@ -70,9 +70,9 @@
 
    内存: 630 GB
 
-3. PaddlePaddle 版本：2.3.1
+3. PaddlePaddle 版本：2.4rc
 
-4. PaddleNLP 版本：2.3.5 (develop)
+4. PaddleNLP 版本：2.4.3
 
 5. 评估设置
 
@@ -82,13 +82,14 @@
   - 微调
 
     ```
+    cd ../
     python train.py --dataset_dir "./data/" --save_dir "./checkpoints" --max_seq_length 128 --model_name "ernie-3.0-base-zh" --batch_size 8 --learning_rate 3e-5 --epochs 100 --logging_steps 5 --early_stop --early_stop_num 10
     ```
 
   - 提示学习
 
     ```
-    python train.py --data_dir ./data/ --output_dir ./checkpoints/ --prompt "这句话包含的要素有" --model_name_or_path ernie-3.0-base-zh --max_seq_length 128  --learning_rate 3e-5 --ppt_learning_rate 3e-4 --do_train --do_eval --num_train_epochs 100 --logging_steps 5 --per_device_eval_batch_size 32 --per_device_train_batch_size 8 --do_predict --metric_for_best_model macro_f1_score --load_best_model_at_end --evaluation_strategy epoch --save_strategy epoch
+    python train.py --data_dir ./data/ --output_dir ./checkpoints/ --prompt "这句话包含的要素有" --model_name_or_path ernie-3.0-base-zh --max_seq_length 128  --learning_rate 3e-5 --ppt_learning_rate 3e-4 --do_train --do_eval --num_train_epochs 100 --logging_steps 5 --per_device_eval_batch_size 32 --per_device_train_batch_size 8 --do_predict --metric_for_best_model macro_f1_score --load_best_model_at_end --eval_steps 100 --save_total_limit 1
     ```
 
 6. 精度评价指标：Micro F1分数、Macro F1分数
@@ -96,7 +97,7 @@
    | model_name | 训练方式 | Micro F1分数 | Macro F1分数 |
    | ---------- | ------- | ----------- | ----------- |
    | ernie-3.0-base-zh | 微调学习 | 0.7419 | 0.5105 |
-   | ernie-3.0-base-zh | 提示学习 | 0.7838 | 0.6985 |
+   | ernie-3.0-base-zh | 提示学习 | 0.7839 | 0.6003 |
 
 <a name="定制训练"></a>
 ## 3.定制训练
@@ -106,10 +107,10 @@
 <a name="运行环境"></a>
 ### 3.1 运行环境
 
-- python >= 3.6
-- paddlepaddle > 2.3（2.4版本发布前推荐安装[develop版本](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/develop/install/pip/linux-pip.html)）
-- paddlenlp >= 2.3.5
-- paddle2onnx >= 1.0.0rc3
+- python >= 3.7
+- paddlepaddle >= 2.4rc
+- paddlenlp >= 2.4.3
+- paddle2onnx >= 1.0.3
 
 <a name="代码结构"></a>
 ### 3.2 代码结构
@@ -223,6 +224,7 @@ python train.py \
 --do_export \
 --num_train_epochs 100 \
 --logging_steps 5 \
+--save_total_limit 1 \
 --per_device_eval_batch_size 32 \
 --per_device_train_batch_size 8 \
 --metric_for_best_model macro_f1_score \
@@ -248,6 +250,7 @@ python -u -m paddle.distributed.launch --gpus 0,1,2,3 train.py \
 --do_export \
 --num_train_epochs 100 \
 --logging_steps 5 \
+--save_total_limit 1 \
 --per_device_eval_batch_size 32 \
 --per_device_train_batch_size 8 \
 --metric_for_best_model macro_f1_score \
@@ -274,6 +277,7 @@ python -u -m paddle.distributed.launch --gpus 0,1,2,3 train.py \
 - `do_export`: 是否在运行结束时将模型导出为静态图，保存路径为`output_dir/export`。
 - `num_train_epochs`: 训练的最大轮数。
 - `max_steps`: 训练的最大步数。此设置将会覆盖`num_train_epochs`。
+- `save_total_limit`: 模型检查点保存数量。
 - `device`: 使用的设备，默认为`gpu`。
 - `eval_steps`: 评估模型的间隔步数。
 - `logging_steps`: 打印日志的间隔步数。
@@ -352,9 +356,9 @@ python infer.py --model_path_prefix checkpoints/export/model --data_dir ./data -
 可配置参数说明：
 
 - `model_path_prefix`: 导出的静态图模型路径及文件前缀。
-- `model_name_or_path`: 内置预训练模型名，或者模型参数配置目录路径，用于加载tokenizer。默认为`ernie-3.0-base-zh`。
+- `model_name`: 内置预训练模型名，用于加载tokenizer。默认为`ernie-3.0-base-zh`。
 - `data_dir`: 待推理数据所在路径，数据应存放在该目录下的`data.txt`文件。
-- `max_seq_length`: 最大句子长度，超过该长度的文本将被截断，不足的以Pad补全。提示文本不会被截断。
+- `max_length`: 最大句子长度，超过该长度的文本将被截断，不足的以Pad补全。提示文本不会被截断。
 - `batch_size`: 每次预测的样本数量。
 - `device`: 选择推理设备，包括`cpu`和`gpu`。默认为`gpu`。
 - `device_id`: 指定GPU设备ID。

applications/text_classification/multi_label/few-shot/infer.py

@@ -14,23 +14,24 @@
 
 import os
 import six
+import json
 import psutil
 import argparse
 
 import numpy as np
 
 from paddlenlp.utils.log import logger
-from paddlenlp.prompt import AutoTemplate, Verbalizer, InputExample
-from paddlenlp.transformers import AutoTokenizer
+from paddlenlp.prompt import AutoTemplate, PromptDataCollatorWithPadding
+from paddlenlp.transformers import AutoTokenizer, AutoModelForMaskedLM
 import paddle2onnx
 import onnxruntime as ort
 
 # yapf: disable
 parser = argparse.ArgumentParser()
 parser.add_argument("--model_path_prefix", type=str, required=True, help="The path prefix of inference model to be used.")
-parser.add_argument("--model_name_or_path", default="ernie-3.0-base-zh", type=str, help="The directory or name of model.")
+parser.add_argument("--model_name", default="ernie-3.0-base-zh", type=str, help="The name of pretrained model.")
 parser.add_argument("--data_dir", default=None, type=str, help="The path to the prediction data, including label.txt and data.txt.")
-parser.add_argument("--max_seq_length", default=128, type=int, help="The maximum total input sequence length after tokenization.")
+parser.add_argument("--max_length", default=128, type=int, help="The maximum total input sequence length after tokenization.")
 parser.add_argument("--use_fp16", action='store_true', help="Whether to use fp16 inference, only takes effect when deploying on gpu.")
 parser.add_argument("--batch_size", default=200, type=int, help="Batch size per GPU/CPU for predicting.")
 parser.add_argument("--num_threads", default=psutil.cpu_count(logical=False), type=int, help="num_threads for cpu.")
@@ -103,11 +104,6 @@ class InferBackend(object):
                                                   'device_id':
                                                   device_id
                                               }])
-        self.input_handles = [
-            self.predictor.get_inputs()[0].name,
-            self.predictor.get_inputs()[1].name,
-            self.predictor.get_inputs()[2].name
-        ]
 
         if device == "gpu":
             try:
@@ -122,27 +118,52 @@ class InferBackend(object):
         logger.info(">>> [InferBackend] Engine Created ...")
 
     def infer(self, input_dict: dict):
-        input_dict = {
-            k: v
-            for k, v in input_dict.items() if k in self.input_handles
-        }
         result = self.predictor.run(None, input_dict)
         return result
 
 
 class MultiLabelPredictor(object):
 
-    def __init__(self, args, label_list):
-        self._label_list = label_list
-        self._tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
-        self._max_seq_length = args.max_seq_length
-        self._batch_size = args.batch_size
-        self.inference_backend = InferBackend(args.model_path_prefix,
-                                              args.device, args.device_id,
-                                              args.use_fp16, args.num_threads)
-        self._template = AutoTemplate.load_from(
-            os.path.dirname(args.model_path_prefix), self._tokenizer,
-            args.max_seq_length)
+    def __init__(self, args):
+        self.args = args
+        self.tokenizer = AutoTokenizer.from_pretrained(args.model_name)
+        self.model = AutoModelForMaskedLM.from_pretrained(args.model_name)
+        self.template, self.labels, self.input_handles = self.post_init()
+        self.collate_fn = PromptDataCollatorWithPadding(
+            self.tokenizer,
+            padding=True,
+            return_tensors="np",
+            return_attention_mask=True)
+
+        self.inference_backend = InferBackend(self.args.model_path_prefix,
+                                              self.args.device,
+                                              self.args.device_id,
+                                              self.args.use_fp16,
+                                              self.args.num_threads)
+
+    def post_init(self):
+        export_path = os.path.dirname(self.args.model_path_prefix)
+        template_path = os.path.join(export_path, "template_config.json")
+        with open(template_path, "r") as fp:
+            prompt = json.load(fp)
+            template = AutoTemplate.create_from(prompt, self.tokenizer,
+                                                self.args.max_length,
+                                                self.model)
+        keywords = template.extract_template_keywords(template.prompt)
+        inputs = [
+            "input_ids", "token_type_ids", "position_ids", "attention_mask",
+            "masked_positions"
+        ]
+        if "soft" in keywords:
+            inputs.append("soft_token_ids")
+        if "encoder" in keywords:
+            inputs.append("encoder_ids")
+        verbalizer_path = os.path.join(export_path, "verbalizer_config.json")
+        with open(verbalizer_path, "r") as fp:
+            label_words = json.load(fp)
+            labels = sorted(list(label_words.keys()))
+
+        return template, labels, inputs
 
     def predict(self, input_data: list):
         encoded_inputs = self.preprocess(input_data)
@@ -155,14 +176,27 @@ class MultiLabelPredictor(object):
         infer_data = self.inference_backend.infer(input_dict)
         return infer_data
 
-    def infer_batch(self, encoded_inputs):
-        num_sample = len(encoded_inputs["input_ids"])
+    def infer_batch(self, inputs):
+        num_sample = len(inputs)
         infer_data = None
         num_infer_data = None
-        for idx in range(0, num_sample, self._batch_size):
-            l, r = idx, idx + self._batch_size
-            keys = encoded_inputs.keys()
-            input_dict = {k: encoded_inputs[k][l:r] for k in keys}
+        for index in range(0, num_sample, self.args.batch_size):
+            left, right = index, index + self.args.batch_size
+            batch_dict = self.collate_fn(inputs[left:right])
+            input_dict = {}
+            for key in self.input_handles:
+                value = batch_dict[key]
+                if key == "attention_mask":
+                    if value.ndim == 2:
+                        value = (1 - value[:, np.newaxis, np.newaxis, :]) * -1e4
+                    elif value.ndim != 4:
+                        raise ValueError(
+                            "Expect attention mask with ndim=2 or 4, but get ndim={}"
+                            .format(value.ndim))
+                    value = value.astype("float32")
+                else:
+                    value = value.astype("int64")
+                input_dict[key] = value
             results = self._infer(input_dict)
             if infer_data is None:
                 infer_data = [[x] for x in results]
@@ -175,16 +209,8 @@ class MultiLabelPredictor(object):
         return infer_data
 
     def preprocess(self, input_data: list):
-        text = [InputExample(text_a=x) for x in input_data]
-        inputs = [self._template.wrap_one_example(x) for x in text]
-        inputs = {
-            "input_ids":
-            np.array([x["input_ids"] for x in inputs], dtype="int64"),
-            "mask_ids":
-            np.array([x["mask_ids"] for x in inputs], dtype="int64"),
-            "soft_token_ids":
-            np.array([x["soft_token_ids"] for x in inputs], dtype="int64")
-        }
+        text = [{"text_a": x} for x in input_data]
+        inputs = [self.template(x) for x in text]
         return inputs
 
     @staticmethod
@@ -197,7 +223,7 @@ class MultiLabelPredictor(object):
         label_ids = np.argwhere(probs > threshold)
         labels = [[] for _ in range(probs.shape[0])]
         for idx, label_id in label_ids:
-            labels[idx].append(self._label_list[label_id])
+            labels[idx].append(self.labels[label_id])
         return {"label": labels}
 
     def printer(self, result, input_data):
@@ -212,12 +238,10 @@ if __name__ == "__main__":
     for arg_name, arg_value in vars(args).items():
         logger.info("{:20}: {}".format(arg_name, arg_value))
 
-    export_path = os.path.dirname(args.model_path_prefix)
-    labels, _ = Verbalizer.load_from(export_path)
+    predictor = MultiLabelPredictor(args)
 
     text_dir = os.path.join(args.data_dir, "data.txt")
     with open(text_dir, "r", encoding="utf-8") as f:
         text_list = [x.strip() for x in f.readlines()]
 
-    predictor = MultiLabelPredictor(args, labels)
     predictor.predict(text_list)

applications/text_classification/multi_label/few-shot/requirements_cpu.txt

@@ -0,0 +1,5 @@
+psutil
+paddlepaddle>=2.4rc
+paddlenlp>=2.4.3
+paddle2onnx>=1.0.3
+onnxruntime

applications/text_classification/multi_label/few-shot/requirements_gpu.txt

@@ -0,0 +1,7 @@
+psutil
+paddlepaddle-gpu>=2.4rc
+paddlenlp>=2.4.3
+paddle2onnx>=1.0.3
+onnxruntime-gpu
+onnx
+onnxconverter-common

applications/text_classification/multi_label/few-shot/train.py

@@ -15,6 +15,7 @@
 from dataclasses import dataclass, field
 import os
 import sys
+from collections import defaultdict
 
 import paddle
 import paddle.nn.functional as F
@@ -41,8 +42,6 @@ from metric import MetricReport
 class DataArguments:
     data_dir: str = field(default="./data", metadata={"help": "The dataset dictionary includes train.txt, dev.txt and label.txt files."})
     prompt: str = field(default=None, metadata={"help": "The input prompt for tuning."})
-    soft_encoder: str = field(default="lstm", metadata={"help": "The encoder type of soft template, `lstm`, `mlp` or None."})
-    encoder_hidden_size: int = field(default=200, metadata={"help": "The dimension of soft embeddings."})
 
 
 @dataclass
@@ -67,17 +66,20 @@ def main():
     tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path)
 
     # Define the template for preprocess and the verbalizer for postprocess.
-    template = AutoTemplate.create_from(
-        data_args.prompt,
-        tokenizer,
-        training_args.max_seq_length,
-        model=model,
-        prompt_encoder=data_args.soft_encoder,
-        encoder_hidden_size=data_args.encoder_hidden_size)
-    logger.info("Using template: {}".format(template.template))
+    template = AutoTemplate.create_from(data_args.prompt,
+                                        tokenizer,
+                                        training_args.max_seq_length,
+                                        model=model)
+    logger.info("Using template: {}".format(template.prompt))
 
     label_file = os.path.join(data_args.data_dir, "label.txt")
-    verbalizer = SoftVerbalizer.from_file(tokenizer, model, label_file)
+    with open(label_file, "r", encoding="utf-8") as fp:
+        label_words = defaultdict(list)
+        for line in fp:
+            data = line.strip().split("==")
+            word = data[1] if len(data) > 1 else data[0].split("##")[-1]
+            label_words[data[0]].append(word)
+    verbalizer = SoftVerbalizer(label_words, tokenizer, model)
 
     # Load the few-shot datasets.
     train_ds, dev_ds, test_ds = load_local_dataset(
@@ -139,11 +141,24 @@ def main():
 
     # Export static model.
     if training_args.do_export:
+        template = prompt_model.template
+        template_keywords = template.extract_template_keywords(template.prompt)
         input_spec = [
-            InputSpec(shape=[None, None], dtype="int64"),  # input_ids
-            InputSpec(shape=[None, None], dtype="int64"),  # mask_ids
-            InputSpec(shape=[None, None], dtype="int64"),  # soft_token_ids
+            InputSpec(shape=[None, None], dtype="int64"),  # input_ids,
+            InputSpec(shape=[None, None], dtype="int64"),  # token_type_ids
+            InputSpec(shape=[None, None], dtype="int64"),  # position_ids
+            InputSpec(shape=[None, None, None, None],
+                      dtype="float32")  # attention_mask
         ]
+        if "mask" in template_keywords:
+            input_spec.append(InputSpec(shape=[None],
+                                        dtype="int64"))  # masked_positions
+        if "soft" in template_keywords:
+            input_spec.append(InputSpec(shape=[None, None],
+                                        dtype="int64"))  # soft_token_ids
+        if "encoder" in template_keywords:
+            input_spec.append(InputSpec(shape=[None, None],
+                                        dtype="int64"))  # encoder_ids
         export_path = os.path.join(training_args.output_dir, 'export')
         trainer.export_model(export_path,
                              input_spec=input_spec,

applications/text_classification/multi_label/few-shot/utils.py

@@ -15,7 +15,6 @@
 import os
 
 from paddlenlp.datasets import load_dataset
-from paddlenlp.prompt import InputExample
 
 
 def load_local_dataset(data_path, splits, label_list):
@@ -39,14 +38,14 @@ def load_local_dataset(data_path, splits, label_list):
             for idx, line in enumerate(fp):
                 data = line.strip().split("\t")
                 if len(data) == 1:
-                    yield InputExample(text_a=data[0])
+                    yield {"text_a": data[0]}
                 else:
                     text, label = data
                     label = label.strip().split(",")
                     label = [
                         float(1) if x in label else float(0) for x in label_list
                     ]
-                    yield InputExample(text_a=text, labels=label)
+                    yield {"text_a": text, "labels": label}
 
     split_map = {"train": "train.txt", "dev": "dev.txt", "test": "test.txt"}
     datasets = []

applications/text_classification/multi_label/retrieval_based/README.md

@@ -0,0 +1,512 @@
+# 基于检索的多标签文本分类方法
+
+ **目录**
+
+* [1. 基于语义索引的多标签分类任务介绍](#基于语义索引的多标签分类任务介绍)
+* [2. 代码结构说明](#代码结构说明)
+* [3. 环境准备](#环境准备)
+* [4. 数据准备](#数据准备)
+* [5. 模型训练](#模型训练)
+* [6. 模型评估](#模型训练)
+* [7. 模型预测](#模型预测)
+* [8. 模型部署](#模型部署)
+* [9. 分类流程](#分类流程)
+
+<a name="基于语义索引的多标签分类任务介绍"></a>
+
+# 1.基于语义索引的多标签分类任务介绍
+
+以前的分类任务中，标签信息作为无实际意义，独立存在的one-hot编码形式存在，这种做法会潜在的丢失标签的语义信息，本方案把文本分类任务中的标签信息转换成含有语义信息的语义向量，将文本分类任务转换成向量检索和匹配的任务。这样做的好处是对于一些类别标签不是很固定的场景，或者需要经常有一些新增类别的需求的情况非常合适。另外，对于一些新的相关的分类任务，这种方法也不需要模型重新学习或者设计一种新的模型结构来适应新的任务。总的来说，这种基于检索的文本分类方法能够有很好的拓展性，能够利用标签里面包含的语义信息，不需要重新进行学习。这种方法可以应用到相似标签推荐，文本标签标注，金融风险事件分类，政务信访分类等领域。
+
+本方案是基于语义索引模型的分类，语义索引模型的目标是：给定输入文本，模型可以从海量候选召回库中**快速、准确**地召回一批语义相关文本。基于语义索引的多标签分类方法有两种，第一种方法是直接把标签变成召回库，即把输入文本和标签的文本进行匹配，第二种是利用召回的文本带有类别标签，把召回文本的类别标签作为给定输入文本的类别。本方案使用双塔模型，训练阶段引入In-batch Negatives  策略，使用hnswlib建立索引库，并把标签作为召回库，进行召回测试。最后利用召回的结果使用 Accuracy 指标来评估语义索引模型的分类的效果。
+
+**注意** 基于语义索引的文本分类的标签在预测过程中会抽取成向量，所以标签需要文本的形式，不能是ID形式的标签。
+
+<a name="代码结构说明"></a>
+
+## 2. 代码结构说明
+
+```
+|—— data.py # 数据读取、数据转换等预处理逻辑
+|—— base_model.py # 语义索引模型基类
+|—— train.py # In-batch Negatives 策略的训练主脚本
+|—— model.py # In-batch Negatives 策略核心网络结构
+
+|—— recall.py # 基于训练好的语义索引模型，从召回库中召回给定文本的相似文本
+|—— evaluate.py # 根据召回结果和评估集计算评估指标
+|—— metric.py # Macro F1和Micro F1评估指标
+|—— predict.py # 给定输入文件，计算文本 pair 的相似度
+|—— export_model.py # 动态图转换成静态图
+|—— export_to_serving.py # 静态图转 Serving
+|—— scripts
+    |—— export_model.sh  # 动态图转换成静态图脚本
+    |—— predict.sh  # 预测 bash 版本
+    |—— evaluate.sh # 评估 bash 版本
+    |—— run_build_index.sh # 构建索引 bash 版本
+    |—— train.sh  # 训练 bash 版本
+    |—— export_to_serving.sh  # Paddle Inference 转 Serving 的 bash 脚本
+    |—— run.sh # 构建Milvus向量的 bash 版本
+|—— utils
+    ├── config.py # Milvus 的配置文件
+    ├── feature_extract.py # 向量抽取文件
+    ├── milvus_util.py # Milvus 的配置文件
+|—— deploy
+    |—— python
+        |—— predict.py # PaddleInference
+        |—— deploy.sh # Paddle Inference 部署脚本
+        |—— rpc_client.py # Paddle Serving 的 Client 端
+        |—— web_service.py # Paddle Serving 的 Serving 端
+        |—— config_nlp.yml # Paddle Serving 的配置文件
+
+```
+
+<a name="环境准备"></a>
+
+## 3. 环境准备
+
+推荐使用GPU进行训练，在预测阶段使用CPU或者GPU均可。
+
+**环境依赖**
+* python >= 3.7
+* paddlepaddle >= 2.3.1
+* paddlenlp >= 2.3.4
+* hnswlib >= 0.5.2
+* visualdl >= 2.2.2
+
+```
+pip install -r requirements.txt
+```
+
+<a name="数据准备"></a>
+
+## 4. 数据准备
+
+训练需要准备指定格式的本地数据集,如果没有已标注的数据集，可以参考[文本分类任务doccano数据标注使用指南](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/applications/text_classification/doccano.md)进行文本分类数据标注。
+
+**指定格式本地数据集目录结构**
+
+```
+├── data # 数据集目录
+    ├── label.txt # 标签集
+    ├── dev.txt # 验证集
+    ├── test.txt # 测试集
+    ├── train.txt  # 训练集
+```
+
+**训练、开发、测试数据集**
+
+train.txt(训练数据集文件)， dev.txt(开发数据集文件)，test.txt(可选，测试数据集文件)，文件中文本与标签类别名用tab符`'\t'`分隔开，由于文本有多个标签，需要把文本拆分成多个单标签的形式，即多行文本标签对。训练集指用于训练模型的数据；开发集指用于评测模型表现的数据，可以根据模型在开发集上的精度调整训练参数和模型；测试集用于测试模型表现，没有测试集时可以使用开发集代替。
+
+- train.txt/test.txt 文件格式：
+```text
+<文本>'\t'<标签1>
+<文本>'\t'<标签2>
+...
+```
+- train.txt/test.txt 文件样例：
+```text
+茄子怎么做才好吃？怎么做才好吃？	生活
+茄子怎么做才好吃？怎么做才好吃？	美食/烹饪
+茄子怎么做才好吃？怎么做才好吃？	烹饪方法
+...
+```
+- dev.txt 文件格式：
+```
+<文本>'\t'<标签1>,<标签2>,<标签3>
+<文本>'\t'<标签2>,<标签5>
+```
+- dev.txt 文件样例：
+```text
+克隆人的思想和原体是一样的吗？	教育/科学,理工学科,生物学
+家用燃气灶哪个牌子好些?最好是高效节能的~最好是高效的~	生活,购物
+咨询下，黛莱美面膜怎么样，那里有咨询下，黛莱美怎么样，那里有	生活,美容/塑身,护肤
+...
+```
+
+**分类标签**
+
+label.txt(分类标签文件)记录数据集中所有标签集合，每一行为一个标签名。
+- label.txt 文件格式：
+```text
+<标签>
+<标签>
+...
+```
+- label.txt 文件样例：
+```text
+魔力宝贝
+完美游戏
+地球科学
+美食/烹饪
+肝胆外科
+动漫
+婚嫁
+历史话题
+新生儿
+...
+```
+
+<a name="模型训练"></a>
+
+## 5. 模型训练
+
+我们使用百科知识问答的数据来构建训练集，开发集。
+
+**训练集（train.txt）** 和 **开发集(dev.txt)** 格式一致，训练集30k条，开发集10k条，每行由文本的标题，内容和类别标签组成，以tab符分割，第一列是问题的标题和问题的描述拼接，剩下的列问题的类别。
+**召回库（label.txt）** 类别的数量是323类，召回标签库的构建有2种方式，第一种是把所有的类别标签当成召回库，第二种是把训练集当成召回集合，我们以第一种为例。
+
+数据集选择的是百科问答数据集的一个子集，问答数据集详情请参考[nlp_chinese_corpus](https://github.com/brightmart/nlp_chinese_corpus)
+
+- [baike_qa_category](https://paddlenlp.bj.bcebos.com/applications/baike_qa_multilabel.zip)
+
+```
+wget https://paddlenlp.bj.bcebos.com/applications/baike_qa_multilabel.zip
+unzip  baike_qa_category.zip
+```
+
+### 单机单卡训练/单机多卡训练
+
+这里采用单机多卡方式进行训练，通过如下命令，指定 GPU 0,1 卡;如果采用单机单卡训练，只需要把`--gpus`参数设置成单卡的卡号即可。
+
+如果使用CPU进行训练，则需要吧`--gpus`参数去除，然后吧`device`设置成cpu即可，详细请参考train.sh文件的训练设置
+
+然后运行下面的命令使用GPU训练，得到语义索引模型：
+
+```
+root_path=inbatch
+data_path=data
+python -u -m paddle.distributed.launch --gpus "0,1" \
+    train.py \
+    --device gpu \
+    --save_dir ./checkpoints/${root_path} \
+    --batch_size 24 \
+    --learning_rate 5E-5 \
+    --epochs 100 \
+    --output_emb_size 0 \
+    --save_steps 50 \
+    --max_seq_length 384 \
+    --warmup_proportion 0.0 \
+    --margin 0.2 \
+    --recall_result_dir "recall_result_dir" \
+    --recall_result_file "recall_result.txt" \
+    --train_set_file ${data_path}/train.txt \
+    --corpus_file ${data_path}/label.txt   \
+    --similar_text_pair_file ${data_path}/dev.txt \
+    --evaluate True
+```
+
+参数含义说明
+
+* `device`: 使用 cpu/gpu 进行训练
+* `save_dir`: 模型存储路径
+* `batch_size`: 训练的batch size的大小
+* `learning_rate`: 训练的学习率的大小
+* `epochs`: 训练的epoch数
+* `output_emb_size`: Transformer 顶层输出的文本向量维度
+* `save_steps`： 模型存储 checkpoint 的间隔 steps 个数
+* `max_seq_length`: 输入序列的最大长度
+* `margin`: 正样本相似度与负样本之间的目标 Gap
+* `train_set_file`: 训练集文件
+* `evaluate`: 是否开启边训练边评估模型训练效果，默认开启
+* `recall_result_dir`: 召回结果存储目录
+* `recall_result_file`: 召回结果的文件名
+* `hnsw_m`: hnsw 算法相关参数，保持默认即可
+* `hnsw_ef`: hnsw 算法相关参数，保持默认即可
+* `recall_num`: 对 1 个文本召回的相似文本数量
+* `similar_text_pair`: 由相似文本对构成的评估集
+* `corpus_file`: 召回库数据 corpus_file
+
+也可以使用bash脚本：
+
+```
+sh scripts/train.sh
+```
+
+<a name="模型评估"></a>
+
+## 6. 模型评估
+
+评估脚本命令如下：
+```
+python -u evaluate.py \
+        --similar_text_pair "data/dev.txt" \
+        --recall_result_file "./recall_result_dir/recall_result.txt" \
+        --label_path data/label.txt
+```
+也可以使用bash脚本：
+
+```
+sh scripts/evaluate.sh
+```
+会得到如下的输出结果：
+
+```
+Micro fl score: 99.30934877025769
+Macro f1 score: 78.20694877991563
+```
+
+<a name="模型预测"></a>
+
+## 7. 模型预测
+
+我们可以基于语义索引模型计算文本和标签的语义相似度。
+
+
+### 开始预测
+
+加载训练的语义索引模型，然后计算文本和标签的语义相似度:
+
+```
+root_dir="checkpoints/inbatch/model_best"
+python -u -m paddle.distributed.launch --gpus "0" \
+    predict.py \
+    --device gpu \
+    --params_path "${root_dir}/model_state.pdparams" \
+    --model_name_or_path rocketqa-zh-dureader-query-encoder \
+    --output_emb_size 0 \
+    --batch_size 128 \
+    --max_seq_length 384 \
+    --text_pair_file "data/test.txt"
+```
+
+参数含义说明
+* `device`: 使用 cpu/gpu 进行训练
+* `params_path`： 预训练模型的参数文件名
+* `output_emb_size`: Transformer 顶层输出的文本向量维度
+* `model_name_or_path`: 预训练模型，用于模型和`Tokenizer`的参数初始化。
+* `text_pair_file`: 由文本 Pair 构成的待预测数据集
+
+也可以运行下面的bash脚本：
+
+```
+sh scripts/predict.sh
+```
+predict.sh文件包含了cpu和gpu运行的脚本，默认是gpu运行的脚本
+
+产出如下结果
+```
+0.8841502070426941
+0.7834227681159973
+0.04591505229473114
+0.15116563439369202
+......
+```
+
+<a name="部署"></a>
+
+## 8. 模型部署
+
+模型部署分为：动转静导出，向量引擎，Paddle Inference推理， Paddle Serving服务化这几个部分。为了提升预测速度，通常需要把训练好的模型转换成静态图，然后就可以使用Paddle Inference静态图进行推理，向量引擎则是存放标签的向量的形式，方便快速检索，另外，Paddle Inference可以进一步对模型服务化，即使用Paddle Serving进行服务化，这样可以通过HTTP或者RPC的方式进行调用。Paddle Serving的服务化形式有Pipeline和C++两种形式，Pipeline灵活一点，方便进行修改，C++部署更麻烦一点，但C++的部署形式效率更高。
+
+### 动转静导出
+
+首先把动态图模型转换为静态图：
+
+```
+python export_model.py \
+                 --params_path checkpoints/inbatch/model_best/model_state.pdparams \
+                 --model_name_or_path rocketqa-zh-dureader-query-encoder \
+                 --output_path=./output
+```
+也可以运行下面的bash脚本：
+
+```
+sh scripts/export_model.sh
+```
+### Paddle Inference预测
+
+预测既可以抽取向量也可以计算两个文本的相似度。
+
+修改deploy/python/predict.py中的id2corpus和corpus_list的样本：
+
+```
+# 抽取向量
+id2corpus = {0: {"sentence": "快要到期的“我的资料”怎么续日期？"}}
+# 计算文本和类别的相似度
+corpus_list = [{
+        "sentence": "快要到期的“我的资料”怎么续日期？",
+        'label': '互联网'
+    }, {
+        "sentence": "快要到期的“我的资料”怎么续日期？",
+        'label': '游戏'
+    }]
+
+```
+
+然后使用PaddleInference
+
+```
+python deploy/python/predict.py --model_dir=./output
+```
+也可以运行下面的bash脚本：
+
+```
+sh deploy.sh
+```
+最终输出的是256维度的特征向量和句子对的预测概率：
+
+```
+(1, 768)
+[[ 0.01510728 -0.03822846 -0.0350773  -0.02304687  0.04219331 -0.04335611
+  -0.03983097  0.04164692  0.04074539 -0.02351343  0.04246496 -0.02563381
+   ....
+
+[0.8133385181427002, 0.1509452909231186]
+```
+
+### 向量引擎
+
+模型准备结束以后，开始搭建 Milvus 的向量检索引擎，用于文本语义向量的快速检索，本项目使用[Milvus](https://milvus.io/)开源工具进行向量检索，Milvus 的搭建教程请参考官方教程  [Milvus官方安装教程](https://milvus.io/cn/docs/v1.1.1/milvus_docker-cpu.md)本案例使用的是 Milvus 的1.1.1 CPU版本，建议使用官方的 Docker 安装方式，简单快捷。
+
+
+Milvus 搭建完系统以后就可以插入和检索向量了，首先生成 embedding 向量，每个样本生成768维度的向量：
+
+```
+CUDA_VISIBLE_DEVICES=0 python utils/feature_extract.py \
+        --data_name label \
+        --model_dir ./output \
+        --output_dir data \
+        --corpus_file "./data/label.txt"
+```
+其中 output 目录下存放的是召回的 Paddle Inference 静态图模型。
+
+修改 utils/config.py 的配置 ip 和端口，本项目使用的是8530端口，而 Milvus 默认的是19530，需要根据情况进行修改：
+```
+MILVUS_HOST='your milvus ip'
+MILVUS_PORT = 8530
+```
+
+然后向搭建好的 Milvus 系统插入向量：
+
+```
+python utils/vector_insert.py \
+                    --vector_path ./data/label_embedding.npy
+```
+也可以直接运行：
+
+```bash
+sh scripts/run.sh
+```
+
+### Paddle Serving部署
+
+Paddle Serving 的安装可以参考[Paddle Serving 安装文档](https://github.com/PaddlePaddle/Serving#installation)。需要在服务端和客户端安装相关的依赖，用pip安装Paddle Serving的依赖如下：
+
+```
+pip install paddle-serving-client==0.8.3 -i https://pypi.tuna.tsinghua.edu.cn/simple
+pip install paddle-serving-app==0.8.3 -i https://pypi.tuna.tsinghua.edu.cn/simple
+
+# 如果是CPU部署，只需要安装CPU Server
+pip install paddle-serving-server==0.8.3 -i https://pypi.tuna.tsinghua.edu.cn/simple
+
+# 如果是GPU Server，需要确认环境再选择执行哪一条，推荐使用CUDA 10.2的包
+# CUDA10.2 + Cudnn7 + TensorRT6（推荐）
+pip install paddle-serving-server-gpu==0.8.3.post102 -i https://pypi.tuna.tsinghua.edu.cn/simple
+# CUDA10.1 + TensorRT6
+pip install paddle-serving-server-gpu==0.8.3.post101 -i https://pypi.tuna.tsinghua.edu.cn/simple
+# CUDA11.2 + TensorRT8
+pip install paddle-serving-server-gpu==0.8.3.post112 -i https://pypi.tuna.tsinghua.edu.cn/simple
+```
+更详细的安装信息请参考[链接](https://github.com/PaddlePaddle/Serving/blob/v0.9.0/doc/Install_Linux_Env_CN.md)，安装完依赖后就可以执行下面的步骤。首先把生成的静态图模型导出为 Paddle Serving的格式，命令如下:
+
+```
+python export_to_serving.py \
+    --dirname "output" \
+    --model_filename "inference.get_pooled_embedding.pdmodel" \
+    --params_filename "inference.get_pooled_embedding.pdiparams" \
+    --server_path "./serving_server" \
+    --client_path "./serving_client" \
+    --fetch_alias_names "output_embedding"
+```
+
+参数含义说明
+* `dirname`: 需要转换的模型文件存储路径，Program 结构文件和参数文件均保存在此目录。
+* `model_filename`： 存储需要转换的模型 Inference Program 结构的文件名称。如果设置为 None ，则使用 `__model__` 作为默认的文件名
+* `params_filename`: 存储需要转换的模型所有参数的文件名称。当且仅当所有模型参数被保>存在一个单独的二进制文件中，它才需要被指定。如果模型参数是存储在各自分离的文件中，设置它的值为 None
+* `server_path`: 转换后的模型文件和配置文件的存储路径。默认值为 serving_server
+* `client_path`: 转换后的客户端配置文件存储路径。默认值为 serving_client
+* `fetch_alias_names`: 模型输出的别名设置，比如输入的 input_ids 等，都可以指定成其他名字，默认不指定
+* `feed_alias_names`: 模型输入的别名设置，比如输出 pooled_out 等，都可以重新指定成其他模型，默认不指定
+
+也可以运行下面的 bash 脚本：
+```
+sh scripts/export_to_serving.sh
+```
+
+Paddle Serving的部署采用Pipeline的方式，如果用户有对性能有更高的要求，可以采用C++的部署形式，请参考[Neural Search](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/applications/neural_search/recall/in_batch_negative#c%E7%9A%84%E6%96%B9%E5%BC%8F)：
+
+#### Pipeline方式
+
+启动 Pipeline Server:
+
+```
+cd deploy/python/
+python web_service.py
+```
+
+启动客户端调用 Server, 使用 POST的方式：
+
+向服务端发送 POST 请求示例：
+
+```
+curl -X POST -k http://localhost:8090/ernie/prediction -d '{"key": ["0"], "value": ["{\"sentence\": \"中国农业大学怎么样？可以吗？\"}"]}'
+```
+
+也可以使用 rpc的方式：
+首先修改rpc_client.py中需要预测的样本：
+
+```
+list_data = [{
+    "sentence": "中国农业大学怎么样？可以吗？"
+}]
+```
+然后运行：
+
+```
+python rpc_client.py
+```
+模型的输出为：
+
+```
+PipelineClient::predict pack_data time:1658988633.3673246
+PipelineClient::predict before time:1658988633.3678396
+time to cost :0.014188766479492188 seconds
+['output_embedding']
+(1, 768)
+[[-0.06491912 -0.0133915   0.00937684  0.01285653 -0.02468005  0.03528611
+   0.0623698  -0.06062918  0.02238894 -0.05348937  0.02161925  0.04480227
+   ......
+```
+
+可以看到客户端发送了1条文本，返回这个 embedding 向量
+
+<a name="分类流程"></a>
+
+## 9. 分类流程
+
+为了演示基于检索的文本分类流程，我们使用下面的python脚本来完成整个流程，该分类系统使用了Client Server的模式，即抽取向量的模型部署在服务端，然后启动客户端（Client）端去访问，得到分类的结果。
+
+```
+python run_system.py
+```
+代码内置的测试用例为：
+
+```
+list_data = [{"sentence": "中国农业大学怎么样？可以吗？"}]
+```
+会输出如下的结果：
+
+```
+......
+PipelineClient::predict pack_data time:1658988661.507715
+PipelineClient::predict before time:1658988661.5081818
+Extract feature time to cost :0.02322244644165039 seconds
+Search milvus time cost is 0.06801486015319824 seconds
+{'sentence': '中国农业大学怎么样？可以吗？'} 教育/科学 0.6138255596160889
+{'sentence': '中国农业大学怎么样？可以吗？'} 院校信息 0.9188833236694336
+```
+输出的结果包括特征提取和检索的时间，还包含检索出来文本和对应的标签，通过设定阈值等方式可以得到最终的标签。
+
+## Reference
+
+[1] Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih, Dense Passage Retrieval for Open-Domain Question Answering, Preprint 2020.

applications/text_classification/multi_label/retrieval_based/base_model.py

@@ -0,0 +1,180 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import abc
+import sys
+
+import numpy as np
+import paddle
+import paddle.nn as nn
+import paddle.nn.functional as F
+
+
+class SemanticIndexBase(nn.Layer):
+
+    def __init__(self, pretrained_model, dropout=None, output_emb_size=None):
+        super().__init__()
+        self.ptm = pretrained_model
+        self.dropout = nn.Dropout(dropout if dropout is not None else 0.1)
+
+        # if output_emb_size is not None, then add Linear layer to reduce embedding_size,
+        # we recommend set output_emb_size = 256 considering the trade-off beteween
+        # recall performance and efficiency
+
+        self.output_emb_size = output_emb_size
+        if output_emb_size > 0:
+            weight_attr = paddle.ParamAttr(
+                initializer=paddle.nn.initializer.TruncatedNormal(std=0.02))
+            self.emb_reduce_linear = paddle.nn.Linear(768,
+                                                      output_emb_size,
+                                                      weight_attr=weight_attr)
+
+    def get_pooled_embedding(self,
+                             input_ids,
+                             token_type_ids=None,
+                             position_ids=None,
+                             attention_mask=None):
+        _, cls_embedding = self.ptm(input_ids, token_type_ids, position_ids,
+                                    attention_mask)
+        if self.output_emb_size > 0:
+            cls_embedding = self.emb_reduce_linear(cls_embedding)
+        cls_embedding = self.dropout(cls_embedding)
+        cls_embedding = F.normalize(cls_embedding, p=2, axis=-1)
+        return cls_embedding
+
+    def get_semantic_embedding(self, data_loader):
+        self.eval()
+        with paddle.no_grad():
+            for batch_data in data_loader:
+                input_ids, token_type_ids = batch_data
+                text_embeddings = self.get_pooled_embedding(
+                    input_ids, token_type_ids=token_type_ids)
+                yield text_embeddings
+
+    def cosine_sim(self,
+                   query_input_ids,
+                   title_input_ids,
+                   query_token_type_ids=None,
+                   query_position_ids=None,
+                   query_attention_mask=None,
+                   title_token_type_ids=None,
+                   title_position_ids=None,
+                   title_attention_mask=None):
+
+        query_cls_embedding = self.get_pooled_embedding(query_input_ids,
+                                                        query_token_type_ids,
+                                                        query_position_ids,
+                                                        query_attention_mask)
+
+        title_cls_embedding = self.get_pooled_embedding(title_input_ids,
+                                                        title_token_type_ids,
+                                                        title_position_ids,
+                                                        title_attention_mask)
+
+        cosine_sim = paddle.sum(query_cls_embedding * title_cls_embedding,
+                                axis=-1)
+        return cosine_sim
+
+    @abc.abstractmethod
+    def forward(self):
+        pass
+
+
+class SemanticIndexBaseStatic(nn.Layer):
+
+    def __init__(self, pretrained_model, dropout=None, output_emb_size=None):
+        super().__init__()
+        self.ptm = pretrained_model
+        self.dropout = nn.Dropout(dropout if dropout is not None else 0.1)
+
+        # if output_emb_size is not None, then add Linear layer to reduce embedding_size,
+        # we recommend set output_emb_size = 256 considering the trade-off beteween
+        # recall performance and efficiency
+
+        self.output_emb_size = output_emb_size
+        if output_emb_size > 0:
+            weight_attr = paddle.ParamAttr(
+                initializer=paddle.nn.initializer.TruncatedNormal(std=0.02))
+            self.emb_reduce_linear = paddle.nn.Linear(768,
+                                                      output_emb_size,
+                                                      weight_attr=weight_attr)
+
+    @paddle.jit.to_static(input_spec=[
+        paddle.static.InputSpec(shape=[None, None], dtype='int64'),
+        paddle.static.InputSpec(shape=[None, None], dtype='int64')
+    ])
+    def get_pooled_embedding(self,
+                             input_ids,
+                             token_type_ids=None,
+                             position_ids=None,
+                             attention_mask=None):
+        _, cls_embedding = self.ptm(input_ids, token_type_ids, position_ids,
+                                    attention_mask)
+
+        if self.output_emb_size > 0:
+            cls_embedding = self.emb_reduce_linear(cls_embedding)
+        cls_embedding = self.dropout(cls_embedding)
+        cls_embedding = F.normalize(cls_embedding, p=2, axis=-1)
+
+        return cls_embedding
+
+    def get_semantic_embedding(self, data_loader):
+        self.eval()
+        with paddle.no_grad():
+            for batch_data in data_loader:
+                input_ids, token_type_ids = batch_data
+
+                text_embeddings = self.get_pooled_embedding(
+                    input_ids, token_type_ids=token_type_ids)
+
+                yield text_embeddings
+
+    def cosine_sim(self,
+                   query_input_ids,
+                   title_input_ids,
+                   query_token_type_ids=None,
+                   query_position_ids=None,
+                   query_attention_mask=None,
+                   title_token_type_ids=None,
+                   title_position_ids=None,
+                   title_attention_mask=None):
+
+        query_cls_embedding = self.get_pooled_embedding(query_input_ids,
+                                                        query_token_type_ids,
+                                                        query_position_ids,
+                                                        query_attention_mask)
+
+        title_cls_embedding = self.get_pooled_embedding(title_input_ids,
+                                                        title_token_type_ids,
+                                                        title_position_ids,
+                                                        title_attention_mask)
+
+        cosine_sim = paddle.sum(query_cls_embedding * title_cls_embedding,
+                                axis=-1)
+        return cosine_sim
+
+    def forward(self,
+                input_ids,
+                token_type_ids=None,
+                position_ids=None,
+                attention_mask=None):
+        _, cls_embedding = self.ptm(input_ids, token_type_ids, position_ids,
+                                    attention_mask)
+
+        if self.output_emb_size > 0:
+            cls_embedding = self.emb_reduce_linear(cls_embedding)
+        cls_embedding = self.dropout(cls_embedding)
+        cls_embedding = F.normalize(cls_embedding, p=2, axis=-1)
+
+        return cls_embedding

applications/text_classification/multi_label/retrieval_based/data.py

@@ -0,0 +1,273 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+
+import hnswlib
+import numpy as np
+import paddle
+from paddlenlp.utils.log import logger
+
+
+def build_index(corpus_data_loader, model, output_emb_size, hnsw_max_elements,
+                hnsw_ef, hnsw_m):
+
+    index = hnswlib.Index(space='cosine',
+                          dim=output_emb_size if output_emb_size > 0 else 768)
+
+    # Initializing index
+    # max_elements - the maximum number of elements (capacity). Will throw an exception if exceeded
+    # during insertion of an element.
+    # The capacity can be increased by saving/loading the index, see below.
+    #
+    # ef_construction - controls index search speed/build speed tradeoff
+    #
+    # M - is tightly connected with internal dimensionality of the data. Strongly affects memory consumption (~M)
+    # Higher M leads to higher accuracy/run_time at fixed ef/efConstruction
+    index.init_index(max_elements=hnsw_max_elements,
+                     ef_construction=hnsw_ef,
+                     M=hnsw_m)
+
+    # Controlling the recall by setting ef:
+    # higher ef leads to better accuracy, but slower search
+    index.set_ef(hnsw_ef)
+
+    # Set number of threads used during batch search/construction
+    # By default using all available cores
+    index.set_num_threads(16)
+    logger.info("start build index..........")
+    all_embeddings = []
+    for text_embeddings in model.get_semantic_embedding(corpus_data_loader):
+        all_embeddings.append(text_embeddings.numpy())
+    all_embeddings = np.concatenate(all_embeddings, axis=0)
+    index.add_items(all_embeddings)
+    logger.info("Total index number:{}".format(index.get_current_count()))
+    return index
+
+
+def create_dataloader(dataset,
+                      mode='train',
+                      batch_size=1,
+                      batchify_fn=None,
+                      trans_fn=None):
+    if trans_fn:
+        dataset = dataset.map(trans_fn)
+    shuffle = True if mode == 'train' else False
+    if mode == 'train':
+        batch_sampler = paddle.io.DistributedBatchSampler(dataset,
+                                                          batch_size=batch_size,
+                                                          shuffle=shuffle)
+    else:
+        batch_sampler = paddle.io.BatchSampler(dataset,
+                                               batch_size=batch_size,
+                                               shuffle=shuffle)
+
+    return paddle.io.DataLoader(dataset=dataset,
+                                batch_sampler=batch_sampler,
+                                collate_fn=batchify_fn,
+                                return_list=True)
+
+
+def convert_example(example,
+                    tokenizer,
+                    max_seq_length=512,
+                    pad_to_max_seq_len=False):
+    """
+    Builds model inputs from a sequence.
+    A BERT sequence has the following format:
+    - single sequence: ``[CLS] X [SEP]``
+    Args:
+        example(obj:`list(str)`): The list of text to be converted to ids.
+        tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
+            which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+        max_seq_len(obj:`int`): The maximum total input sequence length after tokenization. 
+            Sequences longer than this will be truncated, sequences shorter will be padded.
+        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.
+    Returns:
+        input_ids(obj:`list[int]`): The list of query token ids.
+        token_type_ids(obj: `list[int]`): List of query sequence pair mask.
+    """
+
+    result = []
+    for key, text in example.items():
+        encoded_inputs = tokenizer(text=text,
+                                   max_seq_len=max_seq_length,
+                                   pad_to_max_seq_len=pad_to_max_seq_len)
+        input_ids = encoded_inputs["input_ids"]
+        token_type_ids = encoded_inputs["token_type_ids"]
+        result += [input_ids, token_type_ids]
+    return result
+
+
+def convert_corpus_example(example,
+                           tokenizer,
+                           max_seq_length=512,
+                           pad_to_max_seq_len=False):
+    """
+    Builds model inputs from a sequence.
+
+    A BERT sequence has the following format:
+
+    - single sequence: ``[CLS] X [SEP]``
+
+    Args:
+        example(obj:`list(str)`): The list of text to be converted to ids.
+        tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
+            which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+        max_seq_len(obj:`int`): The maximum total input sequence length after tokenization. 
+            Sequences longer than this will be truncated, sequences shorter will be padded.
+        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.
+
+    Returns:
+        input_ids(obj:`list[int]`): The list of query token ids.
+        token_type_ids(obj: `list[int]`): List of query sequence pair mask.
+    """
+    result = []
+    for k, v in example.items():
+        encoded_inputs = tokenizer(text=v,
+                                   max_seq_len=max_seq_length,
+                                   pad_to_max_seq_len=pad_to_max_seq_len)
+        input_ids = encoded_inputs["input_ids"]
+        token_type_ids = encoded_inputs["token_type_ids"]
+        result += [input_ids, token_type_ids]
+    return result
+
+
+def convert_label_example(example,
+                          tokenizer,
+                          max_seq_length=512,
+                          pad_to_max_seq_len=False):
+    """
+    Builds model inputs from a sequence.
+
+    A BERT sequence has the following format:
+
+    - single sequence: ``[CLS] X [SEP]``
+
+    Args:
+        example(obj:`list(str)`): The list of text to be converted to ids.
+        tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
+            which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+        max_seq_len(obj:`int`): The maximum total input sequence length after tokenization. 
+            Sequences longer than this will be truncated, sequences shorter will be padded.
+        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.
+
+    Returns:
+        input_ids(obj:`list[int]`): The list of query token ids.
+        token_type_ids(obj: `list[int]`): List of query sequence pair mask.
+    """
+    result = []
+    for k, v in example.items():
+        encoded_inputs = tokenizer(text=v,
+                                   max_seq_len=max_seq_length,
+                                   pad_to_max_seq_len=pad_to_max_seq_len)
+        input_ids = encoded_inputs["input_ids"]
+        token_type_ids = encoded_inputs["token_type_ids"]
+        result += [input_ids, token_type_ids]
+    return result
+
+
+def read_text_pair(data_path):
+    """Reads data."""
+    with open(data_path, 'r', encoding='utf-8') as f:
+        for line in f:
+            data = line.rstrip().split("\t")
+            if (len(data) != 2):
+                print(data)
+                continue
+            yield {'sentence': data[0], 'label': data[1]}
+
+
+# ANN - active learning ------------------------------------------------------
+def get_latest_checkpoint(args):
+    """
+        Return: (latest_checkpint_path, global_step)
+    """
+    if not os.path.exists(args.save_dir):
+        return args.init_from_ckpt, 0
+
+    subdirectories = list(next(os.walk(args.save_dir))[1])
+
+    def valid_checkpoint(checkpoint):
+        chk_path = os.path.join(args.save_dir, checkpoint)
+        scheduler_path = os.path.join(chk_path, "model_state.pdparams")
+        succeed_flag_file = os.path.join(chk_path, "succeed_flag_file")
+        return os.path.exists(scheduler_path) and os.path.exists(
+            succeed_flag_file)
+
+    trained_steps = [int(s) for s in subdirectories if valid_checkpoint(s)]
+
+    if len(trained_steps) > 0:
+        return os.path.join(args.save_dir, str(max(trained_steps)),
+                            "model_state.pdparams"), max(trained_steps)
+
+    return args.init_from_ckpt, 0
+
+
+# ANN - active learning ------------------------------------------------------
+def get_latest_ann_data(ann_data_dir):
+    if not os.path.exists(ann_data_dir):
+        return None, -1
+
+    subdirectories = list(next(os.walk(ann_data_dir))[1])
+
+    def valid_checkpoint(step):
+        ann_data_file = os.path.join(ann_data_dir, step, "new_ann_data")
+        # succed_flag_file is an empty file that indicates ann data has been generated
+        succeed_flag_file = os.path.join(ann_data_dir, step,
+                                         "succeed_flag_file")
+        return os.path.exists(succeed_flag_file) and os.path.exists(
+            ann_data_file)
+
+    ann_data_steps = [int(s) for s in subdirectories if valid_checkpoint(s)]
+
+    if len(ann_data_steps) > 0:
+        latest_ann_data_file = os.path.join(ann_data_dir,
+                                            str(max(ann_data_steps)),
+                                            "new_ann_data")
+        logger.info(
+            "Using lateset ann_data_file:{}".format(latest_ann_data_file))
+        return latest_ann_data_file, max(ann_data_steps)
+
+    logger.info("no new ann_data, return (None, -1)")
+    return None, -1
+
+
+def label2ids(label_path):
+    label2id = {}
+    with open(label_path) as f:
+        for idx, label in enumerate(f.readlines()):
+            label = label.strip()
+            label2id[label] = idx
+    return label2id
+
+
+def gen_id2corpus(corpus_file):
+    id2corpus = {}
+    with open(corpus_file, 'r', encoding='utf-8') as f:
+        for idx, line in enumerate(f):
+            id2corpus[idx] = line.rstrip()
+    return id2corpus
+
+
+def gen_text_file(similar_text_pair_file):
+    text2similar_text = {}
+    texts = []
+    with open(similar_text_pair_file, 'r', encoding='utf-8') as f:
+        for idx, line in enumerate(f):
+            splited_line = line.rstrip().split("\t")
+            text, similar_text = splited_line[0], ','.join(splited_line[1:])
+            text2similar_text[text] = similar_text
+            texts.append({"text": text})
+    return texts, text2similar_text

applications/text_classification/multi_label/retrieval_based/deploy/python/config_nlp.yml

@@ -0,0 +1,34 @@
+# worker_num, 最大并发数。当build_dag_each_worker=True时, 框架会创建worker_num个进程，每个进程内构建grpcSever和DAG
+# 当build_dag_each_worker=False时，框架会设置主线程grpc线程池的max_workers=worker_num
+worker_num: 20
+# build_dag_each_worker, False，框架在进程内创建一条DAG；True，框架会每个进程内创建多个独立的DAG
+build_dag_each_worker: false
+
+dag:
+  # op资源类型, True, 为线程模型；False，为进程模型
+  is_thread_op: False
+  # 使用性能分析, True，生成Timeline性能数据，对性能有一定影响；False为不使用
+  tracer:
+    interval_s: 10
+# http端口, rpc_port和http_port不允许同时为空。当rpc_port可用且http_port为空时，不自动生成http_port
+http_port: 8090
+# rpc端口, rpc_port和http_port不允许同时为空。当rpc_port为空且http_port不为空时，会自动将rpc_port设置为http_port+1
+rpc_port: 8080
+op:
+  ernie:
+    # 并发数，is_thread_op=True时，为线程并发；否则为进程并发
+    concurrency: 1
+    # 当op配置没有server_endpoints时，从local_service_conf读取本地服务配置
+    local_service_conf:
+      # client类型，包括brpc, grpc和local_predictor.local_predictor不启动Serving服务，进程内预测
+      client_type: local_predictor
+      #ir_optim
+      ir_optim: True
+      # device_type, 0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
+      device_type: 1
+      # 计算硬件ID，当devices为""或不写时为CPU预测；当devices为"0", "0,1,2"时为GPU预测，表示使用的GPU卡
+      devices: '0'
+      # Fetch结果列表，以client_config中fetch_var的alias_name为准, 如果没有设置则全部返回
+      fetch_list: ['output_embedding']
+      # 模型路径
+      model_config: ../../serving_server/

applications/text_classification/multi_label/retrieval_based/deploy/python/deploy.sh

@@ -1,15 +1,15 @@
 # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
-#
+# 
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from ..core_tokenizers.postprocessors import *
+python predict.py --model_dir=../../output

applications/text_classification/multi_label/retrieval_based/deploy/python/predict.py

@@ -0,0 +1,285 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import os
+import sys
+
+import numpy as np
+from scipy.special import softmax
+from scipy import spatial
+import paddle
+from paddle import inference
+from paddlenlp.transformers import AutoModel, AutoTokenizer
+from paddlenlp.data import Stack, Tuple, Pad
+from paddlenlp.datasets import load_dataset
+from paddlenlp.utils.log import logger
+
+sys.path.append('.')
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--model_dir", type=str, required=True,
+    help="The directory to static model.")
+
+parser.add_argument("--max_seq_length", default=128, type=int,
+    help="The maximum total input sequence length after tokenization. Sequences "
+    "longer than this will be truncated, sequences shorter will be padded.")
+parser.add_argument("--batch_size", default=15, type=int,
+    help="Batch size per GPU/CPU for training.")
+parser.add_argument('--device', choices=['cpu', 'gpu', 'xpu'], default="gpu",
+    help="Select which device to train model, defaults to gpu.")
+
+parser.add_argument('--use_tensorrt', default=False, type=eval, choices=[True, False],
+    help='Enable to use tensorrt to speed up.')
+parser.add_argument("--precision", default="fp32", type=str, choices=["fp32", "fp16", "int8"],
+    help='The tensorrt precision.')
+
+parser.add_argument('--cpu_threads', default=10, type=int,
+    help='Number of threads to predict when using cpu.')
+parser.add_argument('--enable_mkldnn', default=False, type=eval, choices=[True, False],
+    help='Enable to use mkldnn to speed up when using cpu.')
+
+parser.add_argument("--save_log_path", type=str, default="./log_output/",
+    help="The file path to save log.")
+args = parser.parse_args()
+# yapf: enable
+
+
+def convert_example(example,
+                    tokenizer,
+                    max_seq_length=512,
+                    pad_to_max_seq_len=False):
+    """
+    Builds model inputs from a sequence.
+    A BERT sequence has the following format:
+    - single sequence: ``[CLS] X [SEP]``
+    Args:
+        example(obj:`list(str)`): The list of text to be converted to ids.
+        tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
+            which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+        max_seq_len(obj:`int`): The maximum total input sequence length after tokenization. 
+            Sequences longer than this will be truncated, sequences shorter will be padded.
+        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.
+    Returns:
+        input_ids(obj:`list[int]`): The list of query token ids.
+        token_type_ids(obj: `list[int]`): List of query sequence pair mask.
+    """
+
+    result = []
+    for key, text in example.items():
+        encoded_inputs = tokenizer(text=text,
+                                   max_seq_len=max_seq_length,
+                                   pad_to_max_seq_len=pad_to_max_seq_len)
+        input_ids = encoded_inputs["input_ids"]
+        token_type_ids = encoded_inputs["token_type_ids"]
+        result += [input_ids, token_type_ids]
+    return result
+
+
+def convert_query_example(example,
+                          tokenizer,
+                          max_seq_length=512,
+                          pad_to_max_seq_len=False):
+    """
+    Builds model inputs from a sequence.
+
+    A BERT sequence has the following format:
+
+    - single sequence: ``[CLS] X [SEP]``
+
+    Args:
+        example(obj:`list(str)`): The list of text to be converted to ids.
+        tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
+            which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+        max_seq_len(obj:`int`): The maximum total input sequence length after tokenization. 
+            Sequences longer than this will be truncated, sequences shorter will be padded.
+        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.
+
+    Returns:
+        input_ids(obj:`list[int]`): The list of query token ids.
+        token_type_ids(obj: `list[int]`): List of query sequence pair mask.
+    """
+    result = []
+    encoded_inputs = tokenizer(text=example['sentence'],
+                               max_seq_len=max_seq_length,
+                               pad_to_max_seq_len=pad_to_max_seq_len,
+                               truncation_strategy="longest_first")
+    input_ids = encoded_inputs["input_ids"]
+    token_type_ids = encoded_inputs["token_type_ids"]
+    result += [input_ids, token_type_ids]
+    return result
+
+
+class Predictor(object):
+
+    def __init__(self,
+                 model_dir,
+                 device="gpu",
+                 max_seq_length=128,
+                 batch_size=32,
+                 use_tensorrt=False,
+                 precision="fp32",
+                 cpu_threads=10,
+                 enable_mkldnn=False):
+        self.max_seq_length = max_seq_length
+        self.batch_size = batch_size
+
+        model_file = model_dir + "/inference.get_pooled_embedding.pdmodel"
+        params_file = model_dir + "/inference.get_pooled_embedding.pdiparams"
+        if not os.path.exists(model_file):
+            raise ValueError("not find model file path {}".format(model_file))
+        if not os.path.exists(params_file):
+            raise ValueError("not find params file path {}".format(params_file))
+        config = paddle.inference.Config(model_file, params_file)
+
+        if device == "gpu":
+            # set GPU configs accordingly
+            # such as intialize the gpu memory, enable tensorrt
+            config.enable_use_gpu(100, 0)
+            precision_map = {
+                "fp16": inference.PrecisionType.Half,
+                "fp32": inference.PrecisionType.Float32,
+                "int8": inference.PrecisionType.Int8
+            }
+            precision_mode = precision_map[precision]
+
+            if args.use_tensorrt:
+                config.enable_tensorrt_engine(max_batch_size=batch_size,
+                                              min_subgraph_size=30,
+                                              precision_mode=precision_mode)
+        elif device == "cpu":
+            # set CPU configs accordingly,
+            # such as enable_mkldnn, set_cpu_math_library_num_threads
+            config.disable_gpu()
+            if args.enable_mkldnn:
+                # cache 10 different shapes for mkldnn to avoid memory leak
+                config.set_mkldnn_cache_capacity(10)
+                config.enable_mkldnn()
+            config.set_cpu_math_library_num_threads(args.cpu_threads)
+        elif device == "xpu":
+            # set XPU configs accordingly
+            config.enable_xpu(100)
+
+        config.switch_use_feed_fetch_ops(False)
+        self.predictor = paddle.inference.create_predictor(config)
+        self.input_handles = [
+            self.predictor.get_input_handle(name)
+            for name in self.predictor.get_input_names()
+        ]
+        self.output_handle = self.predictor.get_output_handle(
+            self.predictor.get_output_names()[0])
+
+    def extract_embedding(self, data, tokenizer):
+        """
+        Predicts the data labels.
+
+        Args:
+            data (obj:`List(str)`): The batch data whose each element is a raw text.
+            tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
+                which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+
+        Returns:
+            results(obj:`dict`): All the feature vectors.
+        """
+        examples = []
+        for idx, text in data.items():
+            print(text)
+            input_ids, segment_ids = convert_query_example(text, tokenizer)
+            examples.append((input_ids, segment_ids))
+
+        batchify_fn = lambda samples, fn=Tuple(
+            Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype="int64"),  # input
+            Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype="int64"
+                ),  # segment
+        ): fn(samples)
+
+        input_ids, segment_ids = batchify_fn(examples)
+        self.input_handles[0].copy_from_cpu(input_ids)
+        self.input_handles[1].copy_from_cpu(segment_ids)
+        self.predictor.run()
+        logits = self.output_handle.copy_to_cpu()
+        return logits
+
+    def predict(self, data, tokenizer):
+        """
+        Predicts the data labels.
+
+        Args:
+            data (obj:`List(str)`): The batch data whose each element is a raw text.
+            tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
+                which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+
+        Returns:
+            results(obj:`dict`): All the predictions probs.
+        """
+
+        examples = []
+        for idx, text in enumerate(data):
+            input_ids, segment_ids, title_ids, title_segment_ids = convert_example(
+                text, tokenizer)
+
+            examples.append(
+                (input_ids, segment_ids, title_ids, title_segment_ids))
+
+        batchify_fn = lambda samples, fn=Tuple(
+            Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype="int64"),  # input
+            Pad(axis=0, pad_val=tokenizer.pad_token_type_id, dtype="int64"
+                ),  # segment
+            Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype="int64"),  # input
+            Pad(axis=0, pad_val=tokenizer.pad_token_type_id, dtype="int64"
+                ),  # segment
+        ): fn(samples)
+
+        query_ids, query_segment_ids, title_ids, title_segment_ids = batchify_fn(
+            examples)
+        self.input_handles[0].copy_from_cpu(query_ids)
+        self.input_handles[1].copy_from_cpu(query_segment_ids)
+        self.predictor.run()
+        query_logits = self.output_handle.copy_to_cpu()
+
+        self.input_handles[0].copy_from_cpu(title_ids)
+        self.input_handles[1].copy_from_cpu(title_segment_ids)
+        self.predictor.run()
+        title_logits = self.output_handle.copy_to_cpu()
+
+        result = [
+            float(1 - spatial.distance.cosine(arr1, arr2))
+            for arr1, arr2 in zip(query_logits, title_logits)
+        ]
+        return result
+
+
+if __name__ == "__main__":
+    # Define predictor to do prediction.
+    predictor = Predictor(args.model_dir, args.device, args.max_seq_length,
+                          args.batch_size, args.use_tensorrt, args.precision,
+                          args.cpu_threads, args.enable_mkldnn)
+
+    output_emb_size = 256
+    tokenizer = AutoTokenizer.from_pretrained(
+        "rocketqa-zh-dureader-query-encoder")
+    id2corpus = {0: {"sentence": "快要到期的“我的资料”怎么续日期？"}}
+    res = predictor.extract_embedding(id2corpus, tokenizer)
+    print(res.shape)
+    print(res)
+    corpus_list = [{
+        "sentence": "快要到期的“我的资料”怎么续日期？",
+        'label': '互联网'
+    }, {
+        "sentence": "快要到期的“我的资料”怎么续日期？",
+        'label': '游戏'
+    }]
+    res = predictor.predict(corpus_list, tokenizer)
+    print(res)

applications/text_classification/multi_label/retrieval_based/deploy/python/rpc_client.py

@@ -0,0 +1,35 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import time
+
+import numpy as np
+from paddle_serving_server.pipeline import PipelineClient
+
+client = PipelineClient()
+client.connect(['127.0.0.1:8080'])
+
+list_data = [{"sentence": "青岛有什么好一点的国际青旅推荐？离海近一点 外国人多一点 氛围好点的"}]
+feed = {}
+for i, item in enumerate(list_data):
+    feed[str(i)] = str(item)
+
+print(feed)
+start_time = time.time()
+ret = client.predict(feed_dict=feed)
+end_time = time.time()
+print("time to cost :{} seconds".format(end_time - start_time))
+result = np.array(eval(ret.value[0]))
+print(ret.key)
+print(result.shape)
+print(result)

applications/text_classification/multi_label/retrieval_based/deploy/python/web_service.py

@@ -0,0 +1,81 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import numpy as np
+import sys
+
+from paddle_serving_server.web_service import WebService, Op
+
+
+def convert_example(example,
+                    tokenizer,
+                    max_seq_length=512,
+                    pad_to_max_seq_len=False):
+    result = []
+    for text in example:
+        encoded_inputs = tokenizer(text=text['sentence'],
+                                   max_seq_len=max_seq_length,
+                                   pad_to_max_seq_len=pad_to_max_seq_len)
+        input_ids = encoded_inputs["input_ids"]
+        token_type_ids = encoded_inputs["token_type_ids"]
+        result += [input_ids, token_type_ids]
+    return result
+
+
+class ErnieOp(Op):
+
+    def init_op(self):
+        from paddlenlp.transformers import AutoTokenizer
+        model_name_or_path = 'rocketqa-zh-dureader-query-encoder'
+        self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
+
+    def preprocess(self, input_dicts, data_id, log_id):
+        from paddlenlp.data import Stack, Tuple, Pad
+        (_, input_dict), = input_dicts.items()
+        print("input dict", input_dict)
+        batch_size = len(input_dict.keys())
+        examples = []
+        for i in range(batch_size):
+            example = eval(input_dict[str(i)])
+            input_ids, segment_ids = convert_example([example], self.tokenizer)
+            examples.append((input_ids, segment_ids))
+        batchify_fn = lambda samples, fn=Tuple(
+            Pad(axis=0, pad_val=self.tokenizer.pad_token_id, dtype="int64"
+                ),  # input
+            Pad(axis=0, pad_val=self.tokenizer.pad_token_type_id, dtype="int64"
+                ),  # segment
+        ): fn(samples)
+        input_ids, segment_ids = batchify_fn(examples)
+        feed_dict = {}
+        feed_dict['input_ids'] = input_ids
+        feed_dict['token_type_ids'] = segment_ids
+        return feed_dict, False, None, ""
+
+    def postprocess(self, input_dicts, fetch_dict, data_id, log_id):
+        new_dict = {}
+        new_dict["output_embedding"] = str(
+            fetch_dict["output_embedding"].tolist())
+        return new_dict, None, ""
+
+
+class ErnieService(WebService):
+
+    def get_pipeline_response(self, read_op):
+        ernie_op = ErnieOp(name="ernie", input_ops=[read_op])
+        return ernie_op
+
+
+ernie_service = ErnieService(name="ernie")
+ernie_service.prepare_pipeline_config("config_nlp.yml")
+ernie_service.run_service()

applications/text_classification/multi_label/retrieval_based/evaluate.py

@@ -0,0 +1,75 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import argparse
+import time
+
+import numpy as np
+from tqdm import tqdm
+
+from metric import MetricReport
+from data import label2ids
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--label_path", type=str,
+                    default='data/label.txt', help="The full path of label file")
+parser.add_argument("--recall_result_file", type=str,
+                    default='./recall_result_dir/recall_result.txt', help="The full path of recall result file")
+parser.add_argument("--similar_text_pair", default='data/dev.txt',
+                    help="The full path of similar pair file")
+
+parser.add_argument("--threshold", default=0.5, type=float,
+                    help="The threshold for selection the labels")
+
+args = parser.parse_args()
+# yapf: enable
+
+
+def evaluate(label2id):
+    metric = MetricReport()
+    text2similar = {}
+    # Encoding labels as one hot
+    with open(args.similar_text_pair, 'r', encoding='utf-8') as f:
+        for line in f:
+            text, similar_text = line.rstrip().rsplit("\t", 1)
+            text2similar[text] = np.zeros(len(label2id))
+            # One hot Encoding
+            for label in similar_text.strip().split(','):
+                text2similar[text][label2id[label]] = 1
+    pred_labels = {}
+    # Convert predicted labels into one hot encoding
+    with open(args.recall_result_file, 'r', encoding='utf-8') as f:
+        for index, line in enumerate(f):
+            text_arr = line.rstrip().split("\t")
+            text, labels, cosine_sim = text_arr
+            # One hot Encoding
+            if (text not in pred_labels):
+                pred_labels[text] = np.zeros(len(label2id))
+            if (float(cosine_sim) > args.threshold):
+                for label in labels.split(','):
+                    pred_labels[text][label2id[label]] = float(cosine_sim)
+
+        for text, probs in tqdm(pred_labels.items()):
+            metric.update(probs, text2similar[text])
+
+        micro_f1_score, macro_f1_score = metric.accumulate()
+        print("Micro fl score: {}".format(micro_f1_score * 100))
+        print("Macro f1 score: {}".format(macro_f1_score * 100))
+
+
+if __name__ == "__main__":
+    label2id = label2ids(args.label_path)
+    evaluate(label2id)

applications/text_classification/multi_label/retrieval_based/export_model.py

@@ -0,0 +1,65 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import os
+from functools import partial
+
+import numpy as np
+import paddle
+import paddle.nn.functional as F
+from paddlenlp.data import Stack, Tuple, Pad
+from paddlenlp.transformers import AutoModel, AutoTokenizer
+
+from base_model import SemanticIndexBase, SemanticIndexBaseStatic
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--params_path", type=str, required=True,
+                    default='./checkpoint/model_900/model_state.pdparams', help="The path to model parameters to be loaded.")
+parser.add_argument("--output_path", type=str, default='./output',
+                    help="The path of model parameter in static graph to be saved.")
+parser.add_argument("--output_emb_size", default=0,
+                    type=int, help="output_embedding_size")
+parser.add_argument("--model_name_or_path", default='rocketqa-zh-dureader-query-encoder',
+                    type=str, help='The pretrained model used for training')
+args = parser.parse_args()
+# yapf: enable
+
+if __name__ == "__main__":
+    pretrained_model = AutoModel.from_pretrained(args.model_name_or_path)
+    tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
+    model = SemanticIndexBaseStatic(pretrained_model,
+                                    output_emb_size=args.output_emb_size)
+
+    if args.params_path and os.path.isfile(args.params_path):
+        state_dict = paddle.load(args.params_path)
+        model.set_dict(state_dict)
+        print("Loaded parameters from %s" % args.params_path)
+    else:
+        raise ValueError(
+            "Please set --params_path with correct pretrained model file")
+    model.eval()
+    # Convert to static graph with specific input description
+    model = paddle.jit.to_static(
+        model,
+        input_spec=[
+            paddle.static.InputSpec(shape=[None, None],
+                                    dtype="int64"),  # input_ids
+            paddle.static.InputSpec(shape=[None, None],
+                                    dtype="int64")  # segment_ids
+        ])
+    # Save in static graph model.
+    save_path = os.path.join(args.output_path, "inference")
+    paddle.jit.save(model, save_path)

applications/text_classification/multi_label/retrieval_based/export_to_serving.py

@@ -0,0 +1,49 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+
+import paddle_serving_client.io as serving_io
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--dirname", type=str, required=True,
+                    default='./output', help="Path of saved model files. Program file and parameter files are saved in this directory.")
+parser.add_argument("--model_filename", type=str, required=True,
+                    default='inference.get_pooled_embedding.pdmodel', help="The name of file to load the inference program. If it is None, the default filename __model__ will be used.")
+parser.add_argument("--params_filename", type=str, required=True,
+                    default='inference.get_pooled_embedding.pdiparams', help="The name of file to load all parameters. It is only used for the case that all parameters were saved in a single binary file. If parameters were saved in separate files, set it as None. Default: None.")
+parser.add_argument("--server_path", type=str, default='./serving_server',
+                    help="The path of server parameter in static graph to be saved.")
+parser.add_argument("--client_path", type=str, default='./serving_client',
+                    help="The path of client parameter in static graph to be saved.")
+parser.add_argument("--feed_alias_names", type=str, default=None,
+                    help='set alias names for feed vars, split by comma \',\', you should run --show_proto to check the number of feed vars')
+parser.add_argument("--fetch_alias_names", type=str, default=None,
+                    help='set alias names for feed vars, split by comma \',\', you should run --show_proto to check the number of fetch vars')
+parser.add_argument("--show_proto", type=bool, default=False,
+                    help='If yes, you can preview the proto and then determine your feed var alias name and fetch var alias name.')
+# yapf: enable
+
+if __name__ == "__main__":
+    args = parser.parse_args()
+    serving_io.inference_model_to_serving(
+        dirname=args.dirname,
+        serving_server=args.server_path,
+        serving_client=args.client_path,
+        model_filename=args.model_filename,
+        params_filename=args.params_filename,
+        show_proto=args.show_proto,
+        feed_alias_names=args.feed_alias_names,
+        fetch_alias_names=args.fetch_alias_names)

applications/text_classification/multi_label/retrieval_based/metric.py

@@ -0,0 +1,85 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import numpy as np
+from sklearn.metrics import f1_score, classification_report
+from paddle.metric import Metric
+from paddlenlp.utils.log import logger
+
+
+class MetricReport(Metric):
+    """
+    F1 score for multi-label text classification task.
+    """
+
+    def __init__(self, name='MetricReport', average='micro'):
+        super(MetricReport, self).__init__()
+        self.average = average
+        self._name = name
+        self.reset()
+
+    def reset(self):
+        """
+        Resets all of the metric state.
+        """
+        self.y_prob = None
+        self.y_true = None
+
+    def f1_score(self, y_prob):
+        """
+        Compute micro f1 score and macro f1 score
+        """
+        threshold = 0.5
+        self.y_pred = y_prob > threshold
+        micro_f1_score = f1_score(y_pred=self.y_pred,
+                                  y_true=self.y_true,
+                                  average='micro')
+        macro_f1_score = f1_score(y_pred=self.y_pred,
+                                  y_true=self.y_true,
+                                  average='macro')
+        return micro_f1_score, macro_f1_score
+
+    def update(self, probs, labels):
+        """
+        Update the probability and label
+        """
+        if self.y_prob is not None:
+            self.y_prob = np.append(self.y_prob, probs, axis=0)
+        else:
+            self.y_prob = probs
+        if self.y_true is not None:
+            self.y_true = np.append(self.y_true, labels, axis=0)
+        else:
+            self.y_true = labels
+
+    def accumulate(self):
+        """
+        Returns micro f1 score and macro f1 score
+        """
+        micro_f1_score, macro_f1_score = self.f1_score(y_prob=self.y_prob)
+        return micro_f1_score, macro_f1_score
+
+    def report(self):
+        """
+        Returns classification report
+        """
+        self.y_pred = self.y_prob > 0.5
+        logger.info("classification report:\n" +
+                    classification_report(self.y_true, self.y_pred, digits=4))
+
+    def name(self):
+        """
+        Returns metric name
+        """
+        return self._name

applications/text_classification/multi_label/retrieval_based/model.py

@@ -0,0 +1,78 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+
+import numpy as np
+import paddle
+import paddle.nn as nn
+import paddle.nn.functional as F
+
+from base_model import SemanticIndexBase
+
+
+class SemanticIndexBatchNeg(SemanticIndexBase):
+
+    def __init__(self,
+                 pretrained_model,
+                 dropout=None,
+                 margin=0.3,
+                 scale=30,
+                 output_emb_size=None):
+        super().__init__(pretrained_model, dropout, output_emb_size)
+
+        self.margin = margin
+        # Used scaling cosine similarity to ease converge
+        self.sacle = scale
+
+    def forward(self,
+                query_input_ids,
+                title_input_ids,
+                query_token_type_ids=None,
+                query_position_ids=None,
+                query_attention_mask=None,
+                title_token_type_ids=None,
+                title_position_ids=None,
+                title_attention_mask=None):
+
+        query_cls_embedding = self.get_pooled_embedding(query_input_ids,
+                                                        query_token_type_ids,
+                                                        query_position_ids,
+                                                        query_attention_mask)
+
+        title_cls_embedding = self.get_pooled_embedding(title_input_ids,
+                                                        title_token_type_ids,
+                                                        title_position_ids,
+                                                        title_attention_mask)
+
+        cosine_sim = paddle.matmul(query_cls_embedding,
+                                   title_cls_embedding,
+                                   transpose_y=True)
+
+        # Substract margin from all positive samples cosine_sim()
+        margin_diag = paddle.full(shape=[query_cls_embedding.shape[0]],
+                                  fill_value=self.margin,
+                                  dtype=paddle.get_default_dtype())
+
+        cosine_sim = cosine_sim - paddle.diag(margin_diag)
+
+        # Scale cosine to ease training converge
+        cosine_sim *= self.sacle
+
+        labels = paddle.arange(0, query_cls_embedding.shape[0], dtype='int64')
+        labels = paddle.reshape(labels, shape=[-1, 1])
+
+        loss = F.cross_entropy(input=cosine_sim, label=labels)
+
+        return loss

applications/text_classification/multi_label/retrieval_based/predict.py

@@ -0,0 +1,118 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from functools import partial
+import argparse
+import sys
+import os
+import random
+import time
+
+import numpy as np
+import paddle
+import paddle.nn.functional as F
+from paddlenlp.transformers import AutoModel, AutoTokenizer
+from paddlenlp.datasets import load_dataset
+from paddlenlp.data import Stack, Tuple, Pad
+
+from data import read_text_pair, convert_example, create_dataloader
+from base_model import SemanticIndexBase
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--text_pair_file", type=str,
+                    required=True, help="The full path of input file")
+parser.add_argument("--params_path", type=str, required=True,
+                    help="The path to model parameters to be loaded.")
+parser.add_argument("--max_seq_length", default=64, type=int, help="The maximum total input sequence length after tokenization. "
+                    "Sequences longer than this will be truncated, sequences shorter will be padded.")
+parser.add_argument("--batch_size", default=32, type=int,
+                    help="Batch size per GPU/CPU for training.")
+parser.add_argument("--output_emb_size", default=None,
+                    type=int, help="output_embedding_size")
+parser.add_argument('--device', choices=['cpu', 'gpu'], default="gpu",
+                    help="Select which device to train model, defaults to gpu.")
+parser.add_argument("--pad_to_max_seq_len", action="store_true",
+                    help="Whether to pad to max seq length.")
+parser.add_argument("--model_name_or_path",default='rocketqa-zh-dureader-query-encoder',type=str,help='The pretrained model used for training')
+args = parser.parse_args()
+# yapf: enable
+
+
+def predict(model, data_loader):
+    """
+    Predicts the data labels.
+
+    Args:
+        model (obj:`SemanticIndexBase`): A model to extract text embedding or calculate similarity of text pair.
+        data_loaer (obj:`List(Example)`): The processed data ids of text pair: [query_input_ids, query_token_type_ids, title_input_ids, title_token_type_ids]
+    Returns:
+        results(obj:`List`): cosine similarity of text pairs.
+    """
+    cosine_sims = []
+    model.eval()
+    with paddle.no_grad():
+        for batch_data in data_loader:
+            query_input_ids, query_token_type_ids, title_input_ids, title_token_type_ids = batch_data
+            batch_cosine_sim = model.cosine_sim(
+                query_input_ids=query_input_ids,
+                title_input_ids=title_input_ids,
+                query_token_type_ids=query_token_type_ids,
+                title_token_type_ids=title_token_type_ids).numpy()
+            cosine_sims.append(batch_cosine_sim)
+        cosine_sims = np.concatenate(cosine_sims, axis=0)
+        return cosine_sims
+
+
+if __name__ == "__main__":
+    paddle.set_device(args.device)
+
+    tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
+    trans_func = partial(convert_example,
+                         tokenizer=tokenizer,
+                         max_seq_length=args.max_seq_length,
+                         pad_to_max_seq_len=args.pad_to_max_seq_len)
+    batchify_fn = lambda samples, fn=Tuple(
+        Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype="int64"
+            ),  # query_input
+        Pad(axis=0, pad_val=tokenizer.pad_token_type_id, dtype="int64"
+            ),  # query_segment
+        Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype="int64"
+            ),  # title_input
+        Pad(axis=0, pad_val=tokenizer.pad_token_type_id, dtype="int64"
+            ),  # tilte_segment
+    ): [data for data in fn(samples)]
+    valid_ds = load_dataset(read_text_pair,
+                            data_path=args.text_pair_file,
+                            lazy=False)
+    valid_data_loader = create_dataloader(valid_ds,
+                                          mode='predict',
+                                          batch_size=args.batch_size,
+                                          batchify_fn=batchify_fn,
+                                          trans_fn=trans_func)
+    pretrained_model = AutoModel.from_pretrained(args.model_name_or_path)
+    model = SemanticIndexBase(pretrained_model,
+                              output_emb_size=args.output_emb_size)
+    if args.params_path and os.path.isfile(args.params_path):
+        state_dict = paddle.load(args.params_path)
+        model.set_dict(state_dict)
+        print("Loaded parameters from %s" % args.params_path)
+    else:
+        raise ValueError(
+            "Please set --params_path with correct pretrained model file")
+    cosin_sim = predict(model, valid_data_loader)
+    for idx, cosine in enumerate(cosin_sim):
+        print('{}'.format(cosine))
+        if (idx > 5):
+            break

applications/text_classification/multi_label/retrieval_based/recall.py

@@ -0,0 +1,135 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# coding=UTF-8
+
+from functools import partial
+import argparse
+import os
+import sys
+import random
+import time
+
+import numpy as np
+import paddle
+import paddle.nn.functional as F
+from paddlenlp.data import Stack, Tuple, Pad
+from paddlenlp.datasets import load_dataset, MapDataset
+from paddlenlp.utils.log import logger
+from paddlenlp.transformers import AutoModel, AutoTokenizer
+
+from base_model import SemanticIndexBase
+from data import convert_corpus_example, create_dataloader
+from data import gen_id2corpus, gen_text_file
+from data import build_index
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--corpus_file", type=str, required=True,
+                    help="The full path of input file")
+parser.add_argument("--similar_text_pair_file", type=str,
+                    required=True, help="The full path of similar text pair file")
+parser.add_argument("--recall_result_dir", type=str, default='recall_result',
+                    help="The full path of recall result file to save")
+parser.add_argument("--recall_result_file", type=str,
+                    default='recall_result_file', help="The file name of recall result")
+parser.add_argument("--params_path", type=str, required=True,
+                    help="The path to model parameters to be loaded.")
+parser.add_argument("--max_seq_length", default=64, type=int, help="The maximum total input sequence length after tokenization. "
+                    "Sequences longer than this will be truncated, sequences shorter will be padded.")
+parser.add_argument("--batch_size", default=32, type=int,
+                    help="Batch size per GPU/CPU for training.")
+parser.add_argument("--output_emb_size", default=None,
+                    type=int, help="output_embedding_size")
+parser.add_argument("--recall_num", default=10, type=int,
+                    help="Recall number for each query from Ann index.")
+parser.add_argument("--hnsw_m", default=100, type=int,
+                    help="Recall number for each query from Ann index.")
+parser.add_argument("--hnsw_ef", default=100, type=int,
+                    help="Recall number for each query from Ann index.")
+parser.add_argument("--hnsw_max_elements", default=1000000,
+                    type=int, help="Recall number for each query from Ann index.")
+parser.add_argument('--device', choices=['cpu', 'gpu'], default="gpu",
+                    help="Select which device to train model, defaults to gpu.")
+parser.add_argument("--model_name_or_path", default='rocketqa-zh-dureader-query-encoder', type=str, help='The pretrained model used for training')
+args = parser.parse_args()
+# yapf: enable
+
+if __name__ == "__main__":
+    paddle.set_device(args.device)
+    rank = paddle.distributed.get_rank()
+    if paddle.distributed.get_world_size() > 1:
+        paddle.distributed.init_parallel_env()
+    tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
+    trans_func = partial(convert_corpus_example,
+                         tokenizer=tokenizer,
+                         max_seq_length=args.max_seq_length)
+    batchify_fn = lambda samples, fn=Tuple(
+        Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype="int64"
+            ),  # text_input
+        Pad(axis=0, pad_val=tokenizer.pad_token_type_id, dtype="int64"
+            ),  # text_segment
+    ): [data for data in fn(samples)]
+    pretrained_model = AutoModel.from_pretrained(args.model_name_or_path)
+    model = SemanticIndexBase(pretrained_model,
+                              output_emb_size=args.output_emb_size)
+    model = paddle.DataParallel(model)
+    # Load pretrained semantic model
+    if args.params_path and os.path.isfile(args.params_path):
+        state_dict = paddle.load(args.params_path)
+        model.set_dict(state_dict)
+        logger.info("Loaded parameters from %s" % args.params_path)
+    else:
+        raise ValueError(
+            "Please set --params_path with correct pretrained model file")
+    id2corpus = gen_id2corpus(args.corpus_file)
+    # conver_example function's input must be dict
+    corpus_list = [{idx: text} for idx, text in id2corpus.items()]
+    corpus_ds = MapDataset(corpus_list)
+    corpus_data_loader = create_dataloader(corpus_ds,
+                                           mode='predict',
+                                           batch_size=args.batch_size,
+                                           batchify_fn=batchify_fn,
+                                           trans_fn=trans_func)
+    # Need better way to get inner model of DataParallel
+    inner_model = model._layers
+    final_index = build_index(corpus_data_loader,
+                              inner_model,
+                              output_emb_size=args.output_emb_size,
+                              hnsw_max_elements=args.hnsw_max_elements,
+                              hnsw_ef=args.hnsw_ef,
+                              hnsw_m=args.hnsw_m)
+    text_list, text2similar_text = gen_text_file(args.similar_text_pair_file)
+    query_ds = MapDataset(text_list)
+    query_data_loader = create_dataloader(query_ds,
+                                          mode='predict',
+                                          batch_size=args.batch_size,
+                                          batchify_fn=batchify_fn,
+                                          trans_fn=trans_func)
+    query_embedding = inner_model.get_semantic_embedding(query_data_loader)
+    if not os.path.exists(args.recall_result_dir):
+        os.mkdir(args.recall_result_dir)
+    recall_result_file = os.path.join(args.recall_result_dir,
+                                      args.recall_result_file)
+    with open(recall_result_file, 'w', encoding='utf-8') as f:
+        for batch_index, batch_query_embedding in enumerate(query_embedding):
+            recalled_idx, cosine_sims = final_index.knn_query(
+                batch_query_embedding.numpy(), args.recall_num)
+            batch_size = len(cosine_sims)
+            for row_index in range(batch_size):
+                text_index = args.batch_size * batch_index + row_index
+                for idx, doc_idx in enumerate(recalled_idx[row_index]):
+                    f.write("{}\t{}\t{}\n".format(
+                        text_list[text_index]["text"], id2corpus[doc_idx],
+                        1.0 - cosine_sims[row_index][idx]))

applications/text_classification/multi_label/retrieval_based/requirements.txt

@@ -0,0 +1,5 @@
+pymilvus==1.1.2
+pandas==0.25.1 
+paddlenlp>=2.3.7    
+hnswlib>=0.5.2
+pybind11

applications/text_classification/multi_label/retrieval_based/run_system.py

@@ -0,0 +1,72 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from functools import partial
+import argparse
+import os
+import sys
+import random
+import time
+
+sys.path.append('utils')
+import numpy as np
+import pandas as pd
+from tqdm import tqdm
+from paddle_serving_server.pipeline import PipelineClient
+
+from data import gen_id2corpus
+from utils.milvus_util import RecallByMilvus
+from utils.config import collection_name, partition_tag
+
+
+def search_in_milvus(text_embedding, corpus_file, query_text):
+    client = RecallByMilvus()
+    start_time = time.time()
+    status, results = client.search(collection_name=collection_name,
+                                    vectors=text_embedding,
+                                    partition_tag=partition_tag)
+    end_time = time.time()
+    print('Search milvus time cost is {} seconds '.format(end_time -
+                                                          start_time))
+    id2corpus = gen_id2corpus(corpus_file)
+    list_data = []
+    for line in results:
+        for item in line:
+            idx = item.id
+            distance = item.distance
+            text = id2corpus[idx]
+            list_data.append([query_text, text, distance])
+    df = pd.DataFrame(list_data,
+                      columns=['query_text', 'label', 'innner_product'])
+    df = df.sort_values(by="innner_product")
+    for index, row in df.iterrows():
+        if (row['innner_product'] > 0.5):
+            print(row['query_text'], row['label'], row['innner_product'])
+
+
+if __name__ == "__main__":
+    client = PipelineClient()
+    client.connect(['127.0.0.1:8080'])
+    corpus_file = "data/label.txt"
+    list_data = [{"sentence": "中国农业大学怎么样？可以吗？"}]
+    feed = {}
+    for i, item in enumerate(list_data):
+        feed[str(i)] = str(item)
+    start_time = time.time()
+    ret = client.predict(feed_dict=feed)
+    end_time = time.time()
+    print("Extract feature time to cost :{} seconds".format(end_time -
+                                                            start_time))
+    result = np.array(eval(ret.value[0]))
+    search_in_milvus(result, corpus_file, list_data[0])

applications/text_classification/multi_label/retrieval_based/scripts/evaluate.sh

@@ -0,0 +1,18 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+python -u evaluate.py \
+        --similar_text_pair "data/dev.txt" \
+        --recall_result_file "./recall_result_dir/recall_result.txt" \
+        --label_path data/label.txt

applications/text_classification/multi_label/retrieval_based/scripts/export_model.sh

@@ -0,0 +1,18 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+python export_model.py \
+                 --params_path checkpoints/inbatch/model_best/model_state.pdparams \
+                 --model_name_or_path rocketqa-zh-dureader-query-encoder \
+                 --output_path=./output 

applications/text_classification/multi_label/retrieval_based/scripts/export_to_serving.sh

@@ -0,0 +1,21 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+python export_to_serving.py \
+    --dirname "output" \
+    --model_filename "inference.get_pooled_embedding.pdmodel" \
+    --params_filename "inference.get_pooled_embedding.pdiparams" \
+    --server_path "serving_server" \
+    --client_path "serving_client" \
+    --fetch_alias_names "output_embedding"

applications/text_classification/multi_label/retrieval_based/scripts/predict.sh

@@ -0,0 +1,26 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# gpu version
+root_dir="checkpoints/inbatch/model_best" 
+python -u -m paddle.distributed.launch --gpus "0" \
+    predict.py \
+    --device gpu \
+    --params_path "${root_dir}/model_state.pdparams" \
+    --model_name_or_path rocketqa-zh-dureader-query-encoder \
+    --output_emb_size 0 \
+    --batch_size 128 \
+    --max_seq_length 384 \
+    --text_pair_file "data/test.txt"
+

applications/text_classification/multi_label/retrieval_based/scripts/run.sh

@@ -0,0 +1,22 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+CUDA_VISIBLE_DEVICES=0 python utils/feature_extract.py \
+        --data_name label \
+        --model_dir ./output \
+        --output_dir data \
+        --corpus_file "./data/label.txt" 
+
+python utils/vector_insert.py \
+                    --vector_path ./data/label_embedding.npy

applications/text_classification/multi_label/retrieval_based/scripts/run_build_index.sh

@@ -0,0 +1,30 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+root_dir="checkpoints/inbatch" 
+python -u -m paddle.distributed.launch --gpus "0" --log_dir "recall_log/" \
+        recall.py \
+        --device gpu \
+        --recall_result_dir "recall_result_dir" \
+        --recall_result_file "recall_result.txt" \
+        --params_path "${root_dir}/model_best/model_state.pdparams" \
+        --model_name_or_path rocketqa-zh-dureader-query-encoder \
+        --hnsw_m 100 \
+        --hnsw_ef 100 \
+        --batch_size 64 \
+        --output_emb_size 0 \
+        --max_seq_length 384 \
+        --recall_num 5 \
+        --similar_text_pair "data/dev.txt" \
+        --corpus_file "data/label.txt" 

applications/text_classification/multi_label/retrieval_based/scripts/train.sh

@@ -0,0 +1,36 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# GPU training
+root_path=inbatch
+data_path=data
+python -u -m paddle.distributed.launch --gpus "0,1" \
+    train.py \
+    --device gpu \
+    --save_dir ./checkpoints/${root_path} \
+    --batch_size 24 \
+    --learning_rate 5E-5 \
+    --epochs 100 \
+    --output_emb_size 0 \
+    --save_steps 50 \
+    --max_seq_length 384 \
+    --warmup_proportion 0.0 \
+    --margin 0.2 \
+    --recall_result_dir "recall_result_dir" \
+    --recall_result_file "recall_result.txt" \
+    --train_set_file ${data_path}/train.txt \
+    --corpus_file ${data_path}/label.txt   \
+    --similar_text_pair_file ${data_path}/dev.txt \
+    --evaluate True \
+    --model_name_or_path rocketqa-zh-dureader-query-encoder

applications/text_classification/multi_label/retrieval_based/train.py

@@ -0,0 +1,300 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import argparse
+import os
+import random
+import time
+import numpy as np
+from functools import partial
+from collections import Counter
+
+import paddle
+import paddle.nn as nn
+from paddlenlp.utils.log import logger
+from paddlenlp.data import Tuple, Pad
+from paddlenlp.datasets import load_dataset, MapDataset
+from paddlenlp.transformers import AutoModel, AutoTokenizer
+from paddlenlp.transformers import LinearDecayWithWarmup
+
+from base_model import SemanticIndexBase
+from model import SemanticIndexBatchNeg
+from data import read_text_pair, convert_example, create_dataloader, gen_id2corpus, gen_text_file, convert_corpus_example
+from data import convert_label_example
+from data import build_index, label2ids
+from metric import MetricReport
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--save_dir", default='./checkpoint', type=str,
+                    help="The output directory where the model checkpoints will be written.")
+parser.add_argument("--max_seq_length", default=512, type=int,
+                    help="The maximum total input sequence length after tokenization. "
+                    "Sequences longer than this will be truncated, sequences shorter will be padded.")
+parser.add_argument("--batch_size", default=32, type=int,
+                    help="Batch size per GPU/CPU for training.")
+parser.add_argument("--output_emb_size", default=256,
+                    type=int, help="output_embedding_size")
+parser.add_argument("--learning_rate", default=5E-5, type=float,
+                    help="The initial learning rate for Adam.")
+parser.add_argument("--weight_decay", default=0.0, type=float,
+                    help="Weight decay if we apply some.")
+parser.add_argument("--epochs", default=10, type=int,
+                    help="Total number of training epochs to perform.")
+parser.add_argument("--warmup_proportion", default=0.0, type=float,
+                    help="Linear warmup proption over the training process.")
+parser.add_argument("--init_from_ckpt", type=str, default=None,
+                    help="The path of checkpoint to be loaded.")
+parser.add_argument("--seed", type=int, default=1000,
+                    help="random seed for initialization")
+parser.add_argument('--device', choices=['cpu', 'gpu'], default="cpu",
+                    help="Select which device to train model, defaults to gpu.")
+parser.add_argument('--save_steps', type=int, default=10000,
+                    help="Inteval steps to save checkpoint")
+parser.add_argument('--log_steps', type=int, default=10,
+                    help="Inteval steps to print log")
+parser.add_argument("--train_set_file", type=str,
+                    default='./data/train.txt',
+                    help="The full path of train_set_file.")
+parser.add_argument("--margin", default=0.2, type=float,
+                    help="Margin beteween pos_sample and neg_samples")
+parser.add_argument("--scale", default=30, type=int,
+                    help="Scale for pair-wise margin_rank_loss")
+parser.add_argument("--corpus_file", type=str, default='./data/label.txt',
+                    help="The full path of input file")
+parser.add_argument("--similar_text_pair_file", type=str,
+                    default='./data/dev.txt',
+                    help="The full path of similar text pair file")
+parser.add_argument("--recall_result_dir", type=str, default='./recall_result_dir',
+                    help="The full path of recall result file to save")
+parser.add_argument("--recall_result_file", type=str,
+                    default='recall_result_init.txt', help="The file name of recall result")
+parser.add_argument("--recall_num", default=50, type=int,
+                    help="Recall number for each query from Ann index.")
+parser.add_argument("--hnsw_m", default=100, type=int,
+                    help="Recall number for each query from Ann index.")
+parser.add_argument("--hnsw_ef", default=100, type=int,
+                    help="Recall number for each query from Ann index.")
+parser.add_argument("--hnsw_max_elements", default=1000000,
+                    type=int, help="Recall number for each query from Ann index.")
+parser.add_argument("--evaluate_result", type=str, default='evaluate_result.txt',
+                    help="evaluate_result")
+parser.add_argument('--evaluate', default=True, type=eval, choices=[True, False],
+                    help='whether evaluate while training')
+parser.add_argument("--model_name_or_path",default='rocketqa-zh-dureader-query-encoder',
+                    type=str,help='The pretrained model used for training')
+parser.add_argument("--threshold", default=0.5, type=float,
+                    help="The threshold for selection the labels")
+args = parser.parse_args()
+# yapf: enable
+
+
+def set_seed(seed):
+    """sets random seed"""
+    random.seed(seed)
+    np.random.seed(seed)
+    paddle.seed(seed)
+
+
+@paddle.no_grad()
+def evaluate(model, corpus_data_loader, query_data_loader, recall_result_file,
+             text_list, id2corpus, label2id):
+    metric = MetricReport()
+    # Load pretrained semantic model
+    inner_model = model._layers
+    final_index = build_index(corpus_data_loader,
+                              inner_model,
+                              output_emb_size=args.output_emb_size,
+                              hnsw_max_elements=args.hnsw_max_elements,
+                              hnsw_ef=args.hnsw_ef,
+                              hnsw_m=args.hnsw_m)
+    query_embedding = inner_model.get_semantic_embedding(query_data_loader)
+    with open(recall_result_file, 'w', encoding='utf-8') as f:
+        for batch_index, batch_query_embedding in enumerate(query_embedding):
+            recalled_idx, cosine_sims = final_index.knn_query(
+                batch_query_embedding.numpy(), args.recall_num)
+            batch_size = len(cosine_sims)
+            for row_index in range(batch_size):
+                text_index = args.batch_size * batch_index + row_index
+                for idx, doc_idx in enumerate(recalled_idx[row_index]):
+                    f.write("{}\t{}\t{}\n".format(
+                        text_list[text_index]["text"], id2corpus[doc_idx],
+                        1.0 - cosine_sims[row_index][idx]))
+    text2similar = {}
+    with open(args.similar_text_pair_file, 'r', encoding='utf-8') as f:
+        for line in f:
+            text_arr = line.rstrip().rsplit("\t")
+            text, similar_text = text_arr[0], text_arr[1]
+            text2similar[text] = np.zeros(len(label2id))
+            # One hot Encoding
+            for label in similar_text.strip().split(','):
+                text2similar[text][label2id[label]] = 1
+    # Convert predicted labels into one hot encoding
+    pred_labels = {}
+    with open(recall_result_file, 'r', encoding='utf-8') as f:
+        for index, line in enumerate(f):
+            text_arr = line.rstrip().split("\t")
+            text, labels, cosine_sim = text_arr
+            # One hot Encoding
+            if (text not in pred_labels):
+                pred_labels[text] = np.zeros(len(label2id))
+            if (float(cosine_sim) > args.threshold):
+                for label in labels.split(','):
+                    pred_labels[text][label2id[label]] = float(cosine_sim)
+
+        for text, probs in pred_labels.items():
+            metric.update(probs, text2similar[text])
+        micro_f1_score, macro_f1_score = metric.accumulate()
+    return macro_f1_score
+
+
+def do_train():
+    paddle.set_device(args.device)
+    rank = paddle.distributed.get_rank()
+    if paddle.distributed.get_world_size() > 1:
+        paddle.distributed.init_parallel_env()
+    set_seed(args.seed)
+    train_ds = load_dataset(read_text_pair,
+                            data_path=args.train_set_file,
+                            lazy=False)
+    pretrained_model = AutoModel.from_pretrained(args.model_name_or_path)
+    tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
+    trans_func = partial(convert_example,
+                         tokenizer=tokenizer,
+                         max_seq_length=args.max_seq_length)
+    batchify_fn = lambda samples, fn=Tuple(
+        Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype='int64'
+            ),  # query_input
+        Pad(axis=0, pad_val=tokenizer.pad_token_type_id, dtype='int64'
+            ),  # query_segment
+        Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype='int64'
+            ),  # title_input
+        Pad(axis=0, pad_val=tokenizer.pad_token_type_id, dtype='int64'
+            ),  # tilte_segment
+    ): [data for data in fn(samples)]
+    train_data_loader = create_dataloader(train_ds,
+                                          mode='train',
+                                          batch_size=args.batch_size,
+                                          batchify_fn=batchify_fn,
+                                          trans_fn=trans_func)
+    model = SemanticIndexBatchNeg(pretrained_model,
+                                  margin=args.margin,
+                                  scale=args.scale,
+                                  output_emb_size=args.output_emb_size)
+    if args.init_from_ckpt and os.path.isfile(args.init_from_ckpt):
+        state_dict = paddle.load(args.init_from_ckpt)
+        model.set_dict(state_dict)
+        print("warmup from:{}".format(args.init_from_ckpt))
+    model = paddle.DataParallel(model)
+    batchify_fn_dev = lambda samples, fn=Tuple(
+        Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype='int64'
+            ),  # text_input
+        Pad(axis=0, pad_val=tokenizer.pad_token_type_id, dtype='int64'
+            ),  # text_segment
+    ): [data for data in fn(samples)]
+    if (args.evaluate):
+        eval_func = partial(convert_example,
+                            tokenizer=tokenizer,
+                            max_seq_length=args.max_seq_length)
+        id2corpus = gen_id2corpus(args.corpus_file)
+        label2id = label2ids(args.corpus_file)
+        # conver_example function's input must be dict
+        corpus_list = [{idx: text} for idx, text in id2corpus.items()]
+        corpus_ds = MapDataset(corpus_list)
+        corpus_data_loader = create_dataloader(corpus_ds,
+                                               mode='predict',
+                                               batch_size=args.batch_size,
+                                               batchify_fn=batchify_fn_dev,
+                                               trans_fn=eval_func)
+        query_func = partial(convert_example,
+                             tokenizer=tokenizer,
+                             max_seq_length=args.max_seq_length)
+        text_list, _ = gen_text_file(args.similar_text_pair_file)
+        query_ds = MapDataset(text_list)
+        query_data_loader = create_dataloader(query_ds,
+                                              mode='predict',
+                                              batch_size=args.batch_size,
+                                              batchify_fn=batchify_fn_dev,
+                                              trans_fn=query_func)
+        if not os.path.exists(args.recall_result_dir):
+            os.mkdir(args.recall_result_dir)
+        recall_result_file = os.path.join(args.recall_result_dir,
+                                          args.recall_result_file)
+    num_training_steps = len(train_data_loader) * args.epochs
+    lr_scheduler = LinearDecayWithWarmup(args.learning_rate, num_training_steps,
+                                         args.warmup_proportion)
+    # Generate parameter names needed to perform weight decay.
+    # All bias and LayerNorm parameters are excluded.
+    decay_params = [
+        p.name for n, p in model.named_parameters()
+        if not any(nd in n for nd in ["bias", "norm"])
+    ]
+    optimizer = paddle.optimizer.AdamW(
+        learning_rate=lr_scheduler,
+        parameters=model.parameters(),
+        weight_decay=args.weight_decay,
+        apply_decay_param_fun=lambda x: x in decay_params,
+        grad_clip=nn.ClipGradByNorm(clip_norm=1.0))
+    global_step = 0
+    best_score = 0.0
+    tic_train = time.time()
+    for epoch in range(1, args.epochs + 1):
+        for step, batch in enumerate(train_data_loader, start=1):
+            query_input_ids, query_token_type_ids, title_input_ids, title_token_type_ids = batch
+            loss = model(query_input_ids=query_input_ids,
+                         title_input_ids=title_input_ids,
+                         query_token_type_ids=query_token_type_ids,
+                         title_token_type_ids=title_token_type_ids)
+            global_step += 1
+            if global_step % args.log_steps == 0 and rank == 0:
+                print(
+                    "global step %d, epoch: %d, batch: %d, loss: %.5f, speed: %.2f step/s"
+                    % (global_step, epoch, step, loss, 10 /
+                       (time.time() - tic_train)))
+                tic_train = time.time()
+            loss.backward()
+            optimizer.step()
+            lr_scheduler.step()
+            optimizer.clear_grad()
+            if not args.evaluate and rank == 0:
+                if global_step % args.save_steps == 0 and rank == 0:
+                    save_dir = os.path.join(args.save_dir,
+                                            "model_%d" % global_step)
+                    if not os.path.exists(save_dir):
+                        os.makedirs(save_dir)
+                    save_param_path = os.path.join(save_dir,
+                                                   'model_state.pdparams')
+                    paddle.save(model.state_dict(), save_param_path)
+                    tokenizer.save_pretrained(save_dir)
+        if args.evaluate and rank == 0:
+            print("evaluating")
+            macro_f1_score = evaluate(model, corpus_data_loader,
+                                      query_data_loader, recall_result_file,
+                                      text_list, id2corpus, label2id)
+            if macro_f1_score > best_score:
+                best_score = macro_f1_score
+                save_dir = os.path.join(args.save_dir, "model_best")
+                if not os.path.exists(save_dir):
+                    os.makedirs(save_dir)
+                save_param_path = os.path.join(save_dir, 'model_state.pdparams')
+                paddle.save(model.state_dict(), save_param_path)
+                tokenizer.save_pretrained(save_dir)
+                with open(os.path.join(save_dir, "train_result.txt"),
+                          'a',
+                          encoding='utf-8') as fp:
+                    fp.write('epoch=%d, global_step: %d, Macro f1: %s\n' %
+                             (epoch, global_step, macro_f1_score))
+
+
+if __name__ == "__main__":
+    do_train()

applications/text_classification/multi_label/retrieval_based/utils/__init__.py

@@ -4,12 +4,10 @@
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
-# http://www.apache.org/licenses/LICENSE-2.0
+#     http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-from ..core_tokenizers.models import *

applications/text_classification/multi_label/retrieval_based/utils/config.py

@@ -0,0 +1,35 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from milvus import MetricType, IndexType
+
+MILVUS_HOST = 'localhost'
+MILVUS_PORT = 8530
+
+output_emb_size = 0
+
+collection_param = {
+    'dimension': output_emb_size if output_emb_size > 0 else 768,
+    'index_file_size': 256,
+    'metric_type': MetricType.IP
+}
+
+index_type = IndexType.FLAT
+index_param = {'nlist': 1000}
+
+top_k = 20
+search_param = {'nprobe': 20}
+
+collection_name = 'multi_label'
+partition_tag = 'partition_2'

applications/text_classification/multi_label/retrieval_based/utils/feature_extract.py

@@ -0,0 +1,217 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import os
+import sys
+
+from tqdm import tqdm
+import numpy as np
+from scipy.special import softmax
+
+import paddle
+from paddle import inference
+import paddlenlp as ppnlp
+from paddlenlp.data import Stack, Tuple, Pad
+from paddlenlp.utils.log import logger
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--model_dir", type=str, required=True,
+    help="The directory to static model.")
+
+parser.add_argument("--corpus_file", type=str, required=True,
+    help="The corpus_file path.")
+
+parser.add_argument("--output_dir", type=str, required=True,
+    help="The ouput path.")
+
+parser.add_argument("--max_seq_length", default=64, type=int,
+    help="The maximum total input sequence length after tokenization. Sequences "
+    "longer than this will be truncated, sequences shorter will be padded.")
+parser.add_argument("--batch_size", default=32, type=int,
+    help="Batch size per GPU/CPU for training.")
+parser.add_argument('--device', choices=['cpu', 'gpu', 'xpu'], default="gpu",
+    help="Select which device to train model, defaults to gpu.")
+parser.add_argument("--data_name", type=str, required=True, help="The dataset name.")
+parser.add_argument('--use_tensorrt', default=False, type=eval, choices=[True, False],
+    help='Enable to use tensorrt to speed up.')
+parser.add_argument("--precision", default="fp32", type=str, choices=["fp32", "fp16", "int8"],
+    help='The tensorrt precision.')
+parser.add_argument("--model_name_or_path",default='rocketqa-zh-dureader-query-encoder',type=str,help='The pretrained model used for training')
+parser.add_argument('--cpu_threads', default=10, type=int,
+    help='Number of threads to predict when using cpu.')
+parser.add_argument('--enable_mkldnn', default=False, type=eval, choices=[True, False],
+    help='Enable to use mkldnn to speed up when using cpu.')
+
+args = parser.parse_args()
+# yapf: enable
+
+
+def convert_example(example,
+                    tokenizer,
+                    max_seq_length=512,
+                    pad_to_max_seq_len=False):
+    """
+    Builds model inputs from a sequence.
+        
+    A BERT sequence has the following format:
+    - single sequence: ``[CLS] X [SEP]``
+    Args:
+        example(obj:`list(str)`): The list of text to be converted to ids.
+        tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
+            which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+        max_seq_len(obj:`int`): The maximum total input sequence length after tokenization. 
+            Sequences longer than this will be truncated, sequences shorter will be padded.
+        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.
+    Returns:
+        input_ids(obj:`list[int]`): The list of query token ids.
+        token_type_ids(obj: `list[int]`): List of query sequence pair mask.
+    """
+
+    result = []
+
+    encoded_inputs = tokenizer(text=example,
+                               max_seq_len=max_seq_length,
+                               pad_to_max_seq_len=pad_to_max_seq_len)
+    input_ids = encoded_inputs["input_ids"]
+    token_type_ids = encoded_inputs["token_type_ids"]
+    result += [input_ids, token_type_ids]
+    return result
+
+
+class Predictor(object):
+
+    def __init__(self,
+                 model_dir,
+                 device="gpu",
+                 max_seq_length=128,
+                 batch_size=32,
+                 use_tensorrt=False,
+                 precision="fp32",
+                 cpu_threads=10,
+                 enable_mkldnn=False):
+        self.max_seq_length = max_seq_length
+        self.batch_size = batch_size
+
+        model_file = model_dir + "/inference.get_pooled_embedding.pdmodel"
+        params_file = model_dir + "/inference.get_pooled_embedding.pdiparams"
+        if not os.path.exists(model_file):
+            raise ValueError("not find model file path {}".format(model_file))
+        if not os.path.exists(params_file):
+            raise ValueError("not find params file path {}".format(params_file))
+        config = paddle.inference.Config(model_file, params_file)
+
+        if device == "gpu":
+            # set GPU configs accordingly
+            # such as intialize the gpu memory, enable tensorrt
+            config.enable_use_gpu(100, 0)
+            precision_map = {
+                "fp16": inference.PrecisionType.Half,
+                "fp32": inference.PrecisionType.Float32,
+                "int8": inference.PrecisionType.Int8
+            }
+            precision_mode = precision_map[precision]
+
+            if args.use_tensorrt:
+                config.enable_tensorrt_engine(max_batch_size=batch_size,
+                                              min_subgraph_size=30,
+                                              precision_mode=precision_mode)
+        elif device == "cpu":
+            # set CPU configs accordingly,
+            # such as enable_mkldnn, set_cpu_math_library_num_threads
+            config.disable_gpu()
+            if args.enable_mkldnn:
+                # cache 10 different shapes for mkldnn to avoid memory leak
+                config.set_mkldnn_cache_capacity(10)
+                config.enable_mkldnn()
+            config.set_cpu_math_library_num_threads(args.cpu_threads)
+        elif device == "xpu":
+            # set XPU configs accordingly
+            config.enable_xpu(100)
+
+        config.switch_use_feed_fetch_ops(False)
+        self.predictor = paddle.inference.create_predictor(config)
+        self.input_handles = [
+            self.predictor.get_input_handle(name)
+            for name in self.predictor.get_input_names()
+        ]
+        self.output_handle = self.predictor.get_output_handle(
+            self.predictor.get_output_names()[0])
+
+    def predict(self, data, tokenizer, data_name):
+        """
+        Predicts the data labels.
+        Args:
+            data (obj:`List(str)`): The batch data whose each element is a raw text.
+            tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
+                which contains most of the methods. Users should refer to the superclass for more information regarding methods.
+        Returns:
+            results(obj:`dict`): All the predictions labels.
+        """
+
+        batchify_fn = lambda samples, fn=Tuple(
+            Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype='int64'),  # input
+            Pad(axis=0, pad_val=tokenizer.pad_token_type_id, dtype='int64'
+                ),  # segment
+        ): fn(samples)
+
+        all_embeddings = []
+        examples = []
+        for idx, text in tqdm(data.items()):
+            input_ids, segment_ids = convert_example(
+                text,
+                tokenizer,
+                max_seq_length=self.max_seq_length,
+                pad_to_max_seq_len=True)
+            examples.append((input_ids, segment_ids))
+            if (len(examples) >= self.batch_size):
+                input_ids, segment_ids = batchify_fn(examples)
+                self.input_handles[0].copy_from_cpu(input_ids)
+                self.input_handles[1].copy_from_cpu(segment_ids)
+                self.predictor.run()
+                logits = self.output_handle.copy_to_cpu()
+                all_embeddings.append(logits)
+                examples = []
+
+        if (len(examples) > 0):
+            input_ids, segment_ids = batchify_fn(examples)
+            self.input_handles[0].copy_from_cpu(input_ids)
+            self.input_handles[1].copy_from_cpu(segment_ids)
+            self.predictor.run()
+            logits = self.output_handle.copy_to_cpu()
+            all_embeddings.append(logits)
+
+        all_embeddings = np.concatenate(all_embeddings, axis=0)
+        np.save('./{}/{}_embedding'.format(args.output_dir, data_name),
+                all_embeddings)
+
+
+def read_text(file_path):
+    file = open(file_path)
+    id2corpus = {}
+    for idx, line in enumerate(file.readlines()):
+        id2corpus[idx] = line.rstrip()
+    return id2corpus
+
+
+if __name__ == "__main__":
+    predictor = Predictor(args.model_dir, args.device, args.max_seq_length,
+                          args.batch_size, args.use_tensorrt, args.precision,
+                          args.cpu_threads, args.enable_mkldnn)
+    data_name = args.data_name
+    tokenizer = ppnlp.transformers.ErnieTokenizer.from_pretrained(
+        args.model_name_or_path)
+    id2corpus = read_text(args.corpus_file)
+    predictor.predict(id2corpus, tokenizer, data_name)

applications/text_classification/multi_label/retrieval_based/utils/milvus_util.py

@@ -0,0 +1,114 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from milvus import *
+
+from config import MILVUS_HOST, MILVUS_PORT, collection_param, index_type, index_param
+from config import top_k, search_param
+
+
+class VecToMilvus():
+
+    def __init__(self):
+        self.client = Milvus(host=MILVUS_HOST, port=MILVUS_PORT)
+
+    def has_collection(self, collection_name):
+        try:
+            status, ok = self.client.has_collection(collection_name)
+            return ok
+        except Exception as e:
+            print("Milvus has_table error:", e)
+
+    def creat_collection(self, collection_name):
+        try:
+            collection_param['collection_name'] = collection_name
+            status = self.client.create_collection(collection_param)
+            print(status)
+            return status
+        except Exception as e:
+            print("Milvus create collection error:", e)
+
+    def create_index(self, collection_name):
+        try:
+            status = self.client.create_index(collection_name, index_type,
+                                              index_param)
+            print(status)
+            return status
+        except Exception as e:
+            print("Milvus create index error:", e)
+
+    def has_partition(self, collection_name, partition_tag):
+        try:
+            status, ok = self.client.has_partition(collection_name,
+                                                   partition_tag)
+            return ok
+        except Exception as e:
+            print("Milvus has partition error: ", e)
+
+    def delete_partition(self, collection_name, partition_tag):
+        try:
+            status = self.client.drop_collection(collection_name)
+            return status
+        except Exception as e:
+            print("Milvus has partition error: ", e)
+
+    def create_partition(self, collection_name, partition_tag):
+        try:
+            status = self.client.create_partition(collection_name,
+                                                  partition_tag)
+            print('create partition {} successfully'.format(partition_tag))
+            return status
+        except Exception as e:
+            print('Milvus create partition error: ', e)
+
+    def insert(self, vectors, collection_name, ids=None, partition_tag=None):
+        try:
+            if not self.has_collection(collection_name):
+                self.creat_collection(collection_name)
+                self.create_index(collection_name)
+                print('collection info: {}'.format(
+                    self.client.get_collection_info(collection_name)[1]))
+            if (partition_tag is not None) and (not self.has_partition(
+                    collection_name, partition_tag)):
+                self.create_partition(collection_name, partition_tag)
+            status, ids = self.client.insert(collection_name=collection_name,
+                                             records=vectors,
+                                             ids=ids,
+                                             partition_tag=partition_tag)
+            self.client.flush([collection_name])
+            print(
+                'Insert {} entities, there are {} entities after insert data.'.
+                format(len(ids),
+                       self.client.count_entities(collection_name)[1]))
+            return status, ids
+        except Exception as e:
+            print("Milvus insert error:", e)
+
+
+class RecallByMilvus():
+
+    def __init__(self):
+        self.client = Milvus(host=MILVUS_HOST, port=MILVUS_PORT)
+
+    def search(self, vectors, collection_name, partition_tag=None):
+        try:
+            status, results = self.client.search(
+                collection_name=collection_name,
+                query_records=vectors,
+                top_k=top_k,
+                params=search_param,
+                partition_tag=partition_tag)
+            return status, results
+        except Exception as e:
+            print('Milvus recall error: ', e)

applications/text_classification/multi_label/retrieval_based/utils/vector_insert.py

@@ -0,0 +1,56 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import random
+import argparse
+
+from tqdm import tqdm
+import numpy as np
+
+from milvus_util import VecToMilvus
+from config import collection_name, partition_tag
+
+# yapf: disable
+parser = argparse.ArgumentParser()
+parser.add_argument("--vector_path", type=str, required=True,
+    help="feature file path.")
+
+args = parser.parse_args()
+# yapf: enable
+
+
+def vector_insert(file_path):
+    embeddings = np.load(file_path)
+    print(embeddings.shape)
+    embedding_ids = [i for i in range(embeddings.shape[0])]
+    print(len(embedding_ids))
+    client = VecToMilvus()
+
+    if (client.has_partition(collection_name, partition_tag)):
+        client.delete_partition(collection_name, partition_tag)
+    data_size = len(embedding_ids)
+    batch_size = 50000
+    for i in tqdm(range(0, data_size, batch_size)):
+        cur_end = i + batch_size
+        if (cur_end > data_size):
+            cur_end = data_size
+        batch_emb = embeddings[np.arange(i, cur_end)]
+        status, ids = client.insert(collection_name=collection_name,
+                                    vectors=batch_emb.tolist(),
+                                    ids=embedding_ids[i:i + batch_size],
+                                    partition_tag=partition_tag)
+
+
+if __name__ == "__main__":
+    vector_insert(args.vector_path)

docs/advanced_guide/prompt.md

@@ -22,11 +22,11 @@ Prompt API 提供了这类算法实现的基本模块，支持[PET](https://arxi
 * [如何定义模板](#如何定义模板)
     * [离散型模板](#离散型模板)
     * [连续型模板](#连续型模板)
+    * [前缀连续型模板](#前缀连续型模板)
     * [快速定义模板](#快速定义模板)
 * [如何定义标签词映射](#如何定义标签词映射)
-    * [单掩码映射](#单掩码映射)
-    * [多掩码映射](#多掩码映射)
-    * [标签词映射分类](#标签词映射分类)
+    * [离散型标签词映射](#离散型标签词映射)
+    * [连续型标签词映射](#连续型标签词映射)
 * [快速开始训练](#快速开始训练)
     * [数据准备](#数据准备)
     * [预训练参数准备](#预训练参数准备)
@@ -39,18 +39,28 @@ Prompt API 提供了这类算法实现的基本模块，支持[PET](https://arxi
 
 ## 如何定义模板
 
-**模板**（Template）的功能是在原有输入文本上增加提示语句，从而将原任务转化为 MLM 任务，可以分为离散型和连续型两种。Prompt API 中提供了统一的数据结构来构造不同类型的模板，输入相应格式的**字符串**，通过解析得到对应的输入模板，即字典构成的列表。
+**模板**（Template）的功能是在原有输入文本上增加提示语句，从而将原任务转化为 MLM 任务，可以分为离散型和连续型两种。Prompt API 中提供了统一的数据结构来构造不同类型的模板，输入相应格式的**字符串**，通过解析得到对应的输入模板。模板由不同字段构成，可任意组合。每个字段中的关键字定义了数据文本或者提示文本，即 `input_ids`，属性可定义该字段是否可截断，以及对应的 `position_ids`，`token_type_ids` 等。
 
 ### 离散型模板
 
 离散型模板 `ManualTemplate` 是直接将提示语句与原始输入文本拼接起来，二者的词向量矩阵共享，均为预训练模型学到的词向量矩阵。可用于实现 PET、RGL 等算法。
 
-**模板关键字**
+**模板关键字及属性**
 
-- ``text`` ：数据集中原始输入文本对应的关键字，包括`text_a`和`text_b`。[数据准备](#数据准备)中介绍了如何将自定义数据集转化为统一格式。
-- ``hard`` ：自定义的文本提示语句。
+- ``text`` ：数据集中原始输入文本对应的关键字，例如，`text_a`、`text_b` 和 `content`。
+- ``hard`` ：自定义的提示语句文本。
 - ``mask`` ：待预测词的占位符。
-- ``sep`` ：用于区分不同的句子。`sep`前后的句子对应不同的`token_type_id`。
+    - ``length`` ：定义 ``mask`` 的数量。
+- ``sep`` ：句间的标志符。不同句子的 `token_type_ids` 需使用 `token_type` 属性定义，默认相同。
+- ``options`` ：数据集字典或者文件中的候选标签序列。
+    - ``add_omask`` ：在每个标签前新增 `[O-MASK]` 字符，用于计算候选标签的预测值。支持实现 [UniMC](https://arxiv.org/pdf/2210.08590.pdf) 算法。
+    - ``add_prompt`` ：给每个标签拼接固定的提示文本，标签位置由 `[OPT]` 标记。支持实现 [EFL](https://arxiv.org/pdf/2104.14690.pdf) 算法。
+
+**模版通用属性**
+
+- `position`: 定义当前字段的起始 `position id`。
+- `token_type`: 定义当前字段及后续字段的 `token type id`。
+- `truncate`: 定义当提示和文本总长度超过最大长度时，当前字段是否可截断。可选 `True` 和 `False`。
 
 **模板定义**
 
@@ -64,16 +74,25 @@ Prompt API 提供了这类算法实现的基本模块，支持[PET](https://arxi
 “{'text': 'text_a'}”和“{'text': 'text_b'}”之间的逻辑关系是{'mask'}
 ```
 
+```
+{'options': './data/label.txt'}{'sep'}下边两句话间的逻辑关系是什么？{'text': 'text_a'}{'sep': None, 'token_type': 1}{'text': 'text_b'}
+```
+其中 `label.txt` 为候选标签的本地文件路径，每行一个候选标签，例如
+
+```
+中立
+蕴含
+矛盾
+```
+
 **样本示例**
 
 例如，对于自然语言推理任务，给定样本
 
 ```python
-from paddlenlp.prompt import InputExample
-sample = InputExample(uid=0,
-                      text_a="心里有些生畏,又不知畏惧什么",
-                      text_b="心里特别开心",
-                      labels="矛盾")
+sample = {
+    "text_a": "心里有些生畏,又不知畏惧什么", "text_b": "心里特别开心", "labels": "矛盾"
+}
 ```
 
 按照模板修改拼接后，最终输入模型的文本数据为
@@ -90,17 +109,17 @@ from paddlenlp.prompt import ManualTemplate
 from paddlenlp.transformers import AutoTokenizer
 
 tokenizer = AutoTokenizer.from_pretrained("ernie-3.0-base-zh")
-template = ManualTemplate(tokenizer=tokenizer,
-                          max_seq_length=512,
-                          template="“{'text': 'text_a'}”和“{'text': 'text_b'}”之间的逻辑关系是{'mask'}")
-input_dict = template.wrap_one_example(sample)
+template = ManualTemplate(prompt="“{'text': 'text_a'}”和“{'text': 'text_b'}”之间的逻辑关系是{'mask'}",
+                          tokenizer=tokenizer,
+                          max_length=512)
+input_dict = template(sample)
 ```
 
 其中初始化参数定义如下
 
+- ``prompt`` ：定义提示语句以及与输入文本组合方式的字符串。
 - ``tokenizer`` ：预训练模型的 tokenizer，用于文本编码。
-- ``max_seq_length`` ：定义输入模型文本的最大长度，包括提示部分。当输入长度超过最大长度时，只会截断`text`关键字对应的输入文本，提示部分不做处理。
-- ``template`` ：定义提示语句以及与输入文本组合方式的字符串。
+- ``max_length`` ：定义输入模型文本的最大长度，包括提示部分。
 
 **使用技巧**
 
@@ -115,24 +134,42 @@ input_dict = template.wrap_one_example(sample)
 
 **模板关键字**
 
-- ``text`` ：数据集中原始输入文本对应的关键字，包括`text_a`和`text_b`。[数据准备](#数据准备)中介绍了如何将自定义数据集转化为统一格式。
+- ``text`` ：数据集中原始输入文本对应的关键字，例如，`text_a`和`text_b`。
 - ``hard`` ：自定义的文本提示语句。
 - ``mask`` ：待预测词的占位符。
-- ``sep`` ：用于区分不同的句子。`sep`前后的句子对应不同的`token_type_id`。
-- ``soft`` 表示连续型提示。若值为 ``None`` ，则使用对应数量的随机初始化向量作为提示；若值为文本，则使用对应长度的连续性向量作为提示，并预训练词向量中文本对应的向量进行初始化。
+- ``sep`` ：句间的标志符。不同句子的 `token_type_ids` 需使用 `token_type` 属性定义，默认相同。
+- ``soft`` 表示连续型提示。若值为 ``None`` ，则随机初始化提示向量；若值为文本，则使用文本对应的预训练字向量初始化提示向量。
+    - ``length`` ：定义 ``soft token`` 的数量。若定义文本长度小于该值，超过部分随机初始化。
+    - ``encoder`` ：定义 `soft token` 的编码器类型，可选 `lstm`，`mlp`。默认为 `None`， 不使用编码器。
+    - ``hidden_size`` ：定义编码器的隐藏层维度。默认与预训练词向量维度相同。
+- ``options`` ：数据集字典或者文件中的候选标签序列。
+    - ``add_omask`` ：在每个标签前新增 `[O-MASK]` 字符，用于计算候选标签的预测值。支持实现 [UniMC](https://arxiv.org/pdf/2210.08590.pdf) 算法。
+    - ``add_prompt`` ：给每个标签拼接固定的提示文本，标签位置由 `[OPT]` 标记。支持实现 [EFL](https://arxiv.org/pdf/2104.14690.pdf) 算法。
+
+**模版通用属性**
+
+- `position`: 定义当前字段的起始 `position id`。
+- `token_type`: 定义当前字段及后续字段的 `token type id`。
+- `truncate`: 定义当提示和文本总长度超过最大长度时，当前字段是否可截断。可选 `True` 和 `False`。
 
 **模板定义**
 
 - 定义长度为 1 的连续型提示，随机初始化：
 
 ```python
-"{'soft': None}{'text': 'text_a'}{'sep'}{'text': 'text_b'}"
+"{'soft'}{'text': 'text_a'}{'sep': None, 'token_type': 1}{'text': 'text_b'}"
 ```
 
-- 定义长度为 10 的连续型提示，随机初始化，其中 ``duplicate`` 参数表示连续型提示的长度（仅在随机初始化时有效，即`soft`值为`None`）：
+- 定义长度为 10 的连续型提示，随机初始化，编码器为 `mlp`：
 
 ```python
-"{'text': 'text_a'}{'sep'}{'text': 'text_b'}{'soft': None, `duplicate`:10}{'mask'}"
+"{'text': 'text_a'}{'sep'}{'text': 'text_b'}{'soft': None, 'length':10, 'encoder': 'mlp'}{'mask'}"
+```
+
+- 定义长度为 15 的连续型提示，使用 `请判断` 初始化前三个 soft token，其余随机初始化，编码器为隐藏层维度为 100 的双层 LSTM：
+
+```python
+"{'text': 'text_a'}{'sep'}{'text': 'text_b'}{'soft': '请判断：', 'length': 15, 'encoder': 'lstm', 'hidden_size': 100}{'mask'}"
 ```
 
 - 定义长度为 15 的连续型提示，使用 `"请判断这两个句子间的逻辑关系："` 的预训练词向量逐一进行初始化：
@@ -156,38 +193,98 @@ from paddlenlp.transformers import AutoTokenizer, AutoModelForMaskedLM
 
 model = AutoModelForMaskedLM.from_pretrained("ernie-3.0-base-zh")
 tokenizer = AutoTokenizer.from_pretrained("ernie-3.0-base-zh")
-template = SoftTemplate(tokenizer=tokenizer,
-                        max_seq_length=512,
-                        model=model,
-                        template="{'text': 'text_a'}{'sep'}{'text': 'text_b'}{'soft': '请判断这两个句子间的逻辑关系：'}{'mask'}",
-                        prompt_encoder='lstm',
-                        encoder_hidden_size=200)
+template = SoftTemplate(prompt="{'text': 'text_a'}{'sep'}{'text': 'text_b'}{'soft': '请判断这两个句子间的逻辑关系：'}{'mask'}",
+                        tokenizer=tokenizer,
+                        max_length=512,
+                        word_embeddings=model.get_input_embeddings())
 ```
 
 其中初始化参数定义如下
 
+- ``prompt`` ：定义连续型模板的提示语句、初始化以及与输入文本组合方式的字符串。
 - ``tokenizer`` ：预训练模型的 tokenizer，用于文本编码。
-- ``max_seq_length`` ：定义输入模型文本的最大长度，包括提示部分。当输入长度超过最大长度时，只会截断`text`关键字对应的输入文本，提示部分不做处理。
-- ``model`` : 预训练语言模型，为了取预训练词向量用于连续型提示向量初始化。
-- ``template`` ：定义连续型模板的提示语句、初始化以及与输入文本组合方式的字符串。
-- ``prompt_encoder`` : 连续型提示向量的编码器，可选 ``mlp`` 和 ``lstm``。默认为 ``None`` ，即无编码器，直接使用向量。
-- ``encoder_hidden_size`` : 连续型提示向量的维度。默认为 ``None`` ，即与预训练词向量维度相同。
+- ``max_seq_length`` ：定义输入模型文本的最大长度，包括提示部分。
+- ``word_embeddings`` ：预训练语言模型的词向量，用于连续型提示向量初始化。
+- ``soft_embeddings`` ：连续型提示向量矩阵，可用于不同模板间的连续型参数共享。设置后将覆盖默认连续型向量矩阵。
 
 **使用技巧**
 
 - 对于分类任务，推荐的连续型提示长度一般为10-20。
 - 对于随机初始化的连续性 prompt 向量，通常用比预训练模型微调更大的学习率来更新参数。
 - 与离散型模板相似，连续型模板对初始化参数也比较敏感。自定义提示语句作为连续性 prompt 向量的初始化参数通常比随机初始化效果好。
-- prompt_encoder 为已有论文中的策略，用于建模不同连续型提示向量之间的序列关系。在实际应用中推荐先去掉 prompt_encoder 调整向量初始化。
+- prompt_encoder 为已有论文中的策略，用于建模不同连续型提示向量之间的序列关系。
+
+
+### 前缀连续型模板
+
+`PrefixTemplate` 同样使用了连续型向量作为提示，与 `SoftTemplate` 的不同，该模版的提示向量不仅仅作用于输入层，每层都会有相应的提示向量。可用于实现 P-Tuning 等算法。
+
+**模板关键字**
+
+- ``text`` ：数据集中原始输入文本对应的关键字，例如，`text_a`和`text_b`。
+- ``hard`` ：自定义的文本提示语句。
+- ``mask`` ：待预测词的占位符。
+- ``sep`` ：句间的标志符。不同句子的 `token_type_ids` 需使用 `token_type` 属性定义，默认相同。
+- ``prefix`` 表示连续型提示，该字段**必须**位于模板首位。若值为 ``None`` ，则随机初始化提示向量；若值为文本，则使用文本对应的预训练字向量初始化提示向量。
+    - ``length`` ：定义 ``soft token`` 的数量。若定义文本长度小于该值，超过部分随机初始化。
+    - ``encoder`` ：定义 `soft token` 的编码器类型，可选 `lstm`，`mlp`。默认为 `None`， 不使用编码器。
+    - ``hidden_size`` ：定义编码器的隐藏层维度。默认与预训练词向量维度相同。
+- ``options`` ：数据集字典或者文件中的候选标签序列。
+    - ``add_omask`` ：在每个标签前新增 `[O-MASK]` 字符，用于计算候选标签的预测值。支持实现 [UniMC](https://arxiv.org/pdf/2210.08590.pdf) 算法。
+    - ``add_prompt`` ：给每个标签拼接固定的提示文本，标签位置由 `[OPT]` 标记。支持实现 [EFL](https://arxiv.org/pdf/2104.14690.pdf) 算法。
+
+**模版通用属性**
+
+- `position`: 定义当前字段的起始 `position id`。
+- `token_type`: 定义当前字段及后续字段的 `token type id`。
+- `truncate`: 定义当提示和文本总长度超过最大长度时，当前字段是否可截断。可选 `True` 和 `False`。
+
+**模板定义**
+
+- 定义长度为 15 的连续型提示，随机初始化：
+
+```python
+"{'prefix': '新闻类别', 'length': 10, 'encoder': 'lstm'}{'text': 'text_a'}"
+```
+
+- 定义混合模板，这里`prefix`关键字对应的提示和`hard`对应的提示对应两套不同的向量：
+
+```python
+"{'prefix': '自然语言推理任务：', 'encoder': 'mlp'}{'text': 'text_a'}{'sep'}{'text': 'text_b'}这两个句子间的逻辑关系是{'mask'}"
+```
+
+
+**调用 API**
+
+```python
+from paddlenlp.prompt import PrefixTemplate
+from paddlenlp.transformers import AutoTokenizer, AutoModelForMaskedLM
+
+model = AutoModelForMaskedLM.from_pretrained("ernie-3.0-base-zh")
+tokenizer = AutoTokenizer.from_pretrained("ernie-3.0-base-zh")
+template = PrefixTemplate(prompt="{'prefix': '任务描述'}{'text': 'text_a'}{'mask'}",
+                          tokenizer=tokenizer,
+                          max_length=512,
+                          model=model,
+                          prefix_dropout=0.1)
+```
+
+其中初始化参数定义如下
+
+- ``prompt`` ：定义连续型模板的提示语句、初始化以及与输入文本组合方式的字符串。
+- ``tokenizer`` ：预训练模型的 tokenizer，用于文本编码。
+- ``max_length`` ：定义输入模型文本的最大长度，包括提示部分。
+- ``model`` ：预训练语言模型，用于连续型提示向量初始化，以及根据模型结构生成每层对应的提示向量。
+- ``prefix_dropout`` ：连续型提示向量的丢弃概率，用于正则化。
 
 
 ### 快速定义模板
 
-PaddleNLP 提供了 ``AutoTemplate`` API 以便快速定义单句输入的手工初始化的连续型模板，同时支持直接按照模板类型自动切换离散型模板和离散型模板。
+PaddleNLP 提供了 ``AutoTemplate`` API 快速定义简化离散型模板，也可根据完整模板字符串自动切换 ManualTemplate、SoftTemplate 和 PrefixTemplate。
 
 **模板定义**
 
-- 只定义用于初始化连续型向量的文本提示，即可得到拼接到句尾的连续型模板输入。例如，
+- 快速定义离散型的文本提示。例如，
 
 ```python
 "这篇文章表达了怎样的情感？"
@@ -196,7 +293,7 @@ PaddleNLP 提供了 ``AutoTemplate`` API 以便快速定义单句输入的手工
 等价于
 
 ```python
-"{'text': 'text_a'}{'soft': '这篇文章表达了怎样的情感？'}{'mask'}"
+"{'text': 'text_a'}{'hard': '这篇文章表达了怎样的情感？'}{'mask'}"
 ```
 
 - 当输入为完整模板字符串时，解析得到的模板与[离散型模板](#离散型模板)和[连续型模板](#连续型模板)中描述的一致。
@@ -210,40 +307,37 @@ from paddlenlp.transformers import AutoTokenizer, AutoModelForMaskedLM
 model = AutoModelForMaskedLM.from_pretrained("ernie-3.0-base-zh")
 tokenizer = AutoTokenizer.from_pretrained("ernie-3.0-base-zh")
 # 离散型模板，返回值为 ManualTemplate 实例
-template = AutoTemplate.create_from(template="{'text': 'text_a'}和{'text': 'text_b'}之间的逻辑关系是{'mask'}",
+template = AutoTemplate.create_from(prompt="这个句子表达了怎样的情感？",
                                     tokenizer=tokenizer,
-                                    max_seq_length=512)
+                                    max_length=512)
+
+template = AutoTemplate.create_from(prompt="这个句子表达了怎样的情感？{'text': 'text_a'}{'mask'}",
+                                    tokenizer=tokenizer,
+                                    max_length=512)
 
 # 连续型模板，返回值为 SoftTemplate 实例
-template = AutoTemplate.create_from(template="{'text': 'text_a'}{'sep'}{'text': 'text_b'}{'soft': '请判断这两个句子间的逻辑关系：'}{'mask'}",
+template = AutoTemplate.create_from(prompt="{'text': 'text_a'}{'sep'}{'text': 'text_b'}{'soft': '请判断这两个句子间的逻辑关系：'}{'mask'}",
                                     tokenizer=tokenizer,
-                                    max_seq_length=512,
-                                    model=model,
-                                    prompt_encoder='lstm',
-                                    encoder_hidden_size=200)
+                                    max_length=512,
+                                    model=model)
 
-# 快速定义单句连续型模板，返回值为 SoftTemplate 实例
-template = AutoTemplate.create_from(template="这篇文章表达了怎样的情感？",
+# 前缀连续型模板，返回值为 PrefixTemplate 实例
+template = AutoTemplate.create_from(prompt="{'prefix': None, 'encoder': 'mlp', 'hidden_size': 50}{'text': 'text_a'}",
                                     tokenizer=tokenizer,
-                                    max_seq_length=512,
-                                    model=model,
-                                    prompt_encoder='lstm',
-                                    encoder_hidden_size=200)
+                                    max_length=512,
+                                    model=model)
 ```
 
 其中初始化参数定义如下
 
+- ``prompt`` ：定义离散型/连续型提示、初始化以及和输入文本的组合方式。
 - ``tokenizer`` ：预训练模型的 tokenizer，用于文本编码。
-- ``max_seq_length`` ：定义输入模型文本的最大长度，包括提示部分。当输入长度超过最大长度时，只会截断`text`关键字对应的输入文本，提示部分不做处理。
+- ``max_length`` ：定义输入模型文本的最大长度，包括提示部分。
 - ``model`` ：预训练语言模型，为了取预训练词向量用于连续型提示向量初始化。
-- ``template`` ：定义离散型/连续型提示、初始化以及和输入文本的组合方式。
-- ``prompt_encoder`` ：连续型提示向量的编码器，可选 ``mlp`` 和 ``lstm`` 。默认为 ``None`` ，即无编码器，直接使用向量。
-- ``encoder_hidden_size`` ：连续型提示向量的维度。默认为 ``None`` ，即与预训练词向量维度相同。
-
 
 ## 如何定义标签词映射
 
-**标签词映射**（Verbalizer）也是提示学习中可选的重要模块，用于建立预测词和标签之间的映射，将“预训练-微调”模式中预测标签的任务转换为预测模板中掩码位置的词语，从而将下游任务统一为预训练任务的形式。目前框架支持了离散型标签词映射和 [Word-level Adversarial ReProgramming (WARP)](https://aclanthology.org/2021.acl-long.381/) 方法。
+**标签词映射**（Verbalizer）也是提示学习中可选的重要模块，用于建立预测词和标签之间的映射，将“预训练-微调”模式中预测标签的任务转换为预测模板中掩码位置的词语，从而将下游任务统一为预训练任务的形式。目前框架支持了离散型标签词映射和连续型标签词映射 [Word-level Adversarial ReProgramming (WARP)](https://aclanthology.org/2021.acl-long.381/) 方法。
 
 
 例如，在情感二分类任务中，微调方法和提示学习的标签体系如下
@@ -259,9 +353,9 @@ template = AutoTemplate.create_from(template="这篇文章表达了怎样的情
 具体来说，对于模板 ``{'text':'text_a'}这句话表示我{'mask'}满意。`` ，我们使用映射 ``{'负向': '不', '正向': '很'}`` 将标签 ``负向`` 映射为 ``不`` ，将标签 ``正向`` 映射为 ``很`` 。也就是说，我们期望对于正向情感的文本，预测结果为 ``...这句话表示我很满意。`` ，对于负向情感的文本，预测结果为 ``...这句话表示我不满意。``
 
 
-### 单掩码映射
+### 离散型标签词映射
 
-``ManualVerbalizer`` 支持构造简单的单 ``{'mask'}`` 标签词映射，直接作用于 ``AutoMaskedLM`` 模型结构。当标签对应的预测词长度大于 ``1`` 时取均值。
+``ManualVerbalizer`` 支持构造 ``{'mask'}`` 对应的标签词映射，支持多``{'mask'}``，直接作用于 ``AutoMaskedLM`` 模型结构。当标签对应的预测词长度大于 ``1`` 时，默认取均值。
 
 **调用 API**
 
@@ -271,41 +365,15 @@ from paddlenlp.transformers import AutoTokenizer
 
 tokenizer = AutoTokenizer.from_pretrained("ernie-3.0-base-zh")
 verbalizer = ManualVerbalizer(tokenizer=tokenizer,
-                              labels=['负向', '正向'],
-                              label_words={'负向': '不', '正向': '很'},
-                              prefix=None)
+                              label_words={'负向': '不', '正向': '很'})
 ```
 
 其中初始化参数定义如下
 
+- ``label_words`` : 原标签到预测词之间的映射字典。
 - ``tokenizer`` : 预训练模型的 tokenizer，用于预测词的编码。
-- ``labels`` : 数据集的原标签列表（可选）。
-- ``label_words`` : 原标签到预测词之间的映射字典。如果同时定义了 ``labels`` ，二者的标签集合需要相同。
-- ``prefix`` : 预测词解码前增加的前缀，用于 ``RoBERTa`` 等对前缀敏感的模型，例如 `roberta-large`， `good` 和 ` good` 经过 tokenize 会得到不同的 id。默认为 ``None`` ，无前缀。
 
-
-### 多掩码映射
-
-``MultiMaskVerbalizer`` 继承自 ``ManualVerbalizer`` ，支持多 ``{'mask'}`` 标签词映射。预测词长度需与 ``{'mask'}`` 长度一致。
-
-**调用 API**
-
-```python
-from paddlenlp.prompt import MultiMaskVerbalizer
-from paddlenlp.transformers import AutoTokenizer
-
-tokenizer = AutoTokenizer.from_pretrained("ernie-3.0-base-zh")
-verbalizer = MultiMaskVerbalizer(tokenizer=tokenizer,
-                                 labels=['负向', '正向'],
-                                 label_words={'负向': '生气', '正向': '高兴'},
-                                 prefix=None)
-```
-
-
-其中初始化参数定义同[单掩码映射](#单掩码映射) 。
-
-
-### 标签词映射分类
+### 连续型标签词映射
 
 标签词映射分类器 ``SoftVerbalizer`` 修改了原 ``AutoMaskedLM`` 的模型结构，将预训练模型最后一层“隐藏层-词表”替换为“隐藏层-标签”的映射。该层网络的初始化参数由标签词映射中的预测词词向量来决定，如果预测词长度大于 ``1`` ，则使用词向量均值进行初始化。当前支持的预训练模型包括 ``ErnieForMaskedLM`` 、 ``BertForMaskedLM`` 、 ``AlbertForMaskedLM`` 和 ``RobertaForMaskedLM`` 。可用于实现 WARP 算法。
 
@@ -318,15 +386,13 @@ from paddlenlp.transformers import AutoTokenizer, AutoModelForMaskedLM
 
 model = AutoModelForMaskedLM.from_pretrained("ernie-3.0-base-zh")
 tokenizer = AutoTokenizer.from_pretrained("ernie-3.0-base-zh")
-verbalizer = SoftVerbalizer(tokenizer=tokenizer,
-                            model=model,
-                            labels=['负向', '正向'],
-                            label_words={'负向': '生气', '正向': '高兴'},
-                            prefix=None)
+verbalizer = SoftVerbalizer(label_words={'负向': '生气', '正向': '高兴'},
+                            tokenizer=tokenizer,
+                            model=model)
 ```
 
-其中初始化参数定义同[单掩码映射](#单掩码映射) ，此外
-
+- ``label_words`` : 原标签到预测词之间的映射字典。
+- ``tokenizer`` : 预训练模型的 tokenizer，用于预测词的编码。
 - ``model`` ：预训练语言模型，用于取预训练词向量进行“隐藏层-标签”网络的修改和初始化。
 
 ## 快速开始训练
@@ -335,29 +401,24 @@ verbalizer = SoftVerbalizer(tokenizer=tokenizer,
 
 ### 数据准备
 
-Prompt 框架定义了统一的样本结构 ``InputExample`` 以便进行数据处理，数据集样本需要封装在 ``MapDataset`` 中。
+数据集封装为 ``MapDataset`` 类型。每条数据格式为字典结构，字典中关键字与模板中 `text` 定义的值相对应，统一使用 `labels` 关键字表示样本标签。
 
-例如，对于文本语义相似度 BUSTM 数据集中的原始样本
-
-```python
-data = [
-    {'id': 3, 'sentence1': '你晚上吃了什么', 'sentence2': '你晚上吃啥了', 'label': 1},
-    {'id': 4, 'sentence1': '我想打开滴滴叫的士', 'sentence2': '你叫小欧吗', 'label': 0},
-    {'id': 5, 'sentence1': '女孩子到底是不是你', 'sentence2': '你不是女孩子吗', 'label': 1}
-]
-```
-
-
-需要转换为统一格式
+例如，文本语义相似度 BUSTM 数据集中的数据样本
 
 ```python
 from paddlenlp.datasets import MapDataset
-from paddlenlp.prompt import InputExample
 
-data_ds = MapDataset([InputExample(uid=example["id"],
-                                   text_a=example["sentence1"],
-                                   text_b=example["sentence2"],
-                                   labels=example["label"]) for example in data])
+data_ds = MapDataset([
+    {'id': 3, 'sentence1': '你晚上吃了什么', 'sentence2': '你晚上吃啥了', 'label': 1},
+    {'id': 4, 'sentence1': '我想打开滴滴叫的士', 'sentence2': '你叫小欧吗', 'label': 0},
+    {'id': 5, 'sentence1': '女孩子到底是不是你', 'sentence2': '你不是女孩子吗', 'label': 1}
+])
+
+def convert_label_keyword(input_dict):
+    input_dict["labels"] = input_dict.pop("label")
+    return input_dict
+
+data_ds = data_ds.map(convert_label_keyword)
 ```
 
 ### 预训练参数准备
@@ -383,13 +444,13 @@ from paddlenlp.prompt import ManualVerbalizer
 from paddlenlp.prompt import PromptModelForSequenceClassification
 
 # 定义模板
-template = AutoTemplate.create_from(template="{'text': 'text_a'}和{'text': 'text_b'}说的是{'mask'}同的事情。",
+template = AutoTemplate.create_from(prompt="{'text': 'text_a'}和{'text': 'text_b'}说的是{'mask'}同的事情。",
                                     tokenizer=tokenizer,
-                                    max_seq_length=512)
+                                    max_length=512)
 
 # 定义标签词映射
-verbalizer = ManualVerbalizer(tokenizer=tokenizer,
-                              label_words={0: '不', 1: '相'})
+verbalizer = ManualVerbalizer(label_words={0: '不', 1: '相'},
+                              tokenizer=tokenizer)
 
 # 定义文本分类提示模型
 prompt_model = PromptModelForSequenceClassification(model,
@@ -404,8 +465,8 @@ prompt_model = PromptModelForSequenceClassification(model,
 - ``model`` : 预训练模型实例，支持 ``AutoModelForMaskedLM`` 和 ``AutoModelForSequenceClassification`` 。
 - ``template`` : 模板实例。
 - ``verbalizer`` : 标签词映射实例。当设为 ``None`` 时，不使用标签词映射，模型输出及损失值计算由 ``model`` 类型定义。
-- ``freeze_plm`` : 在训练时是否固定预训练模型参数。对于规模较小的预训练模型，推荐更新预训练模型参数。
-- ``freeze_dropout`` : 在训练时是否固定预训练模型参数并关闭 ``dropout`` 。 当 ``freeze_dropout=True`` ，``freeze_plm`` 也为 ``True`` 。
+- ``freeze_plm`` : 在训练时固定预训练模型参数，默认为 `False`。对于轻量级预训练模型，推荐使用默认值。
+- ``freeze_dropout`` : 在训练时固定预训练模型参数并关闭 ``dropout`` 。 当 ``freeze_dropout=True`` ，``freeze_plm`` 也为 ``True`` 。
 
 
 ### 使用PromptTrainer训练
@@ -497,6 +558,8 @@ if training_args.do_train:
 - WARP: Word-level Adversarial ReProgramming. [[PDF]](https://aclanthology.org/2021.acl-long.381/)
 - RGL: A Simple yet Effective Relation Graph Augmented Prompt-based Tuning Approach for Few-Shot Learning. [[PDF]](https://aclanthology.org/2022.findings-naacl.81/)
 - R-Drop: Regularized Dropout for Neural Networks. [[PDF]](https://arxiv.org/abs/2106.14448)
+- Openprompt: An open-source framework for prompt-learning. [[PDF]](https://arxiv.org/abs/2111.01998)
+
 
 ### 附录
 
@@ -506,9 +569,9 @@ if training_args.do_train:
 
 | 参数              |  类型  | 默认值   |   含义                                                   |
 | ---------------- | ------ | ------- | ------------------------------------------------------- |
-| max_seq_length   |  int   |  512    |    模型输入的最大长度，包括模板部分                          |
-| freeze_plm       |  bool  |  False  |    是否在训练时固定预训练模型的参数                          |
-| freeze_dropout   |  bool  |  False  |    是否在训练时固定预训练模型的参数，同时关闭 dropout         |
+| max_seq_length   |  int   |  512    |   模型输入的最大长度，包括模板部分                          |
+| freeze_plm       |  bool  |  False  |   是否在训练时固定预训练模型的参数                          |
+| freeze_dropout   |  bool  |  False  |   是否在训练时固定预训练模型的参数，同时关闭 dropout         |
 | use_rdrop        |  bool  |  False  |   是否使用 RDrop 策略，详见 [RDrop 论文](https://arxiv.org/abs/2106.14448) |
 | alpha_rdrop      |  float |  5.0    |   RDrop Loss 的权重                                      |
 | use_rgl          |  bool  |  False  |   是否使用 RGL 策略，详见 [RGL 论文](https://aclanthology.org/2022.findings-naacl.81/) |

docs/locale/en/LC_MESSAGES/source/paddlenlp.transformers.ernie.faster_tokenizer.po

@@ -17,7 +17,7 @@ msgstr ""
 "Content-Transfer-Encoding: 8bit\n"
 "Generated-By: Babel 2.10.1\n"
 
-#: ../source/paddlenlp.transformers.ernie.faster_tokenizer.rst:2
-msgid "faster\\_tokenizer"
+#: ../source/paddlenlp.transformers.ernie.fast_tokenizer.rst:2
+msgid "fast\\_tokenizer"
 msgstr ""
 

docs/source/paddlenlp.transformers.bert.fast_tokenizer.rst

@@ -1,7 +1,7 @@
-faster\_tokenizer
+fast\_tokenizer
 ====================================================
 
-.. automodule:: paddlenlp.transformers.bert.faster_tokenizer
+.. automodule:: paddlenlp.transformers.bert.fast_tokenizer
    :members:
    :no-undoc-members:
    :show-inheritance:

docs/source/paddlenlp.transformers.bert.rst

@@ -10,6 +10,6 @@ bert
 .. toctree::
    :maxdepth: 4
 
-   paddlenlp.transformers.bert.faster_tokenizer
+   paddlenlp.transformers.bert.fast_tokenizer
    paddlenlp.transformers.bert.modeling
    paddlenlp.transformers.bert.tokenizer

docs/source/paddlenlp.transformers.ernie.fast_tokenizer.rst

@@ -1,7 +1,7 @@
-faster\_tokenizer
+fast\_tokenizer
 =====================================================
 
-.. automodule:: paddlenlp.transformers.ernie.faster_tokenizer
+.. automodule:: paddlenlp.transformers.ernie.fast_tokenizer
    :members:
    :no-undoc-members:
    :show-inheritance:

docs/source/paddlenlp.transformers.ernie.rst

@@ -10,6 +10,6 @@ ernie
 .. toctree::
    :maxdepth: 4
 
-   paddlenlp.transformers.ernie.faster_tokenizer
+   paddlenlp.transformers.ernie.fast_tokenizer
    paddlenlp.transformers.ernie.modeling
    paddlenlp.transformers.ernie.tokenizer

docs/source/paddlenlp.transformers.ernie_m.faster_tokenizer.rst

@@ -1,7 +1,7 @@
-faster\_tokenizer
+fast\_tokenizer
 ========================================================
 
-.. automodule:: paddlenlp.transformers.ernie_m.faster_tokenizer
+.. automodule:: paddlenlp.transformers.ernie_m.fast_tokenizer
    :members:
    :no-undoc-members:
    :show-inheritance:

docs/source/paddlenlp.transformers.ernie_m.rst

@@ -10,6 +10,6 @@ ernie\_m
 .. toctree::
    :maxdepth: 4
 
-   paddlenlp.transformers.ernie_m.faster_tokenizer
+   paddlenlp.transformers.ernie_m.fast_tokenizer
    paddlenlp.transformers.ernie_m.modeling
    paddlenlp.transformers.ernie_m.tokenizer

docs/source/paddlenlp.transformers.tinybert.fast_tokenizer.rst

@@ -1,7 +1,7 @@
-faster\_tokenizer
+fast\_tokenizer
 ========================================================
 
-.. automodule:: paddlenlp.transformers.tinybert.faster_tokenizer
+.. automodule:: paddlenlp.transformers.tinybert.fast_tokenizer
    :members:
    :no-undoc-members:
    :show-inheritance:

docs/source/paddlenlp.transformers.tinybert.rst

@@ -10,6 +10,6 @@ tinybert
 .. toctree::
    :maxdepth: 4
 
-   paddlenlp.transformers.tinybert.faster_tokenizer
+   paddlenlp.transformers.tinybert.fast_tokenizer
    paddlenlp.transformers.tinybert.modeling
    paddlenlp.transformers.tinybert.tokenizer

docs/trainer.md

@@ -395,6 +395,37 @@ Trainer 是一个简单，但功能完整的 Paddle训练和评估模块，并
 
                         The value of initial scale_loss for fp16. (default: 32768)
 
+  --sharding
+                        是否使用Paddle的Sharding数据并行功能，用户的参数。支持sharding `stage1`, `stage2` or `stage3`。
+                        其中`stage2``stage3`可以和`offload`组合使用。
+                        每个种策略分别为：
+                            stage1 : optimizer 中的参数切分到不同卡
+                            stage2 : optimizer  + gradient 中的参数切分到不同卡
+                            stage3 : parameter + gradient + optimizer  中的参数都切分到不同卡
+                            offload ： offload parameters to cpu 部分参数存放到cpu中
+                         (`str`,  可选, 默认为 `` 不使用sharding)
+                         注意：当前stage3暂时不可用
+
+                        Whether or not to use Paddle Sharding Data Parallel training (in distributed training
+                        only). The base option should be `stage1`, `stage2` or `stage3` and you can add
+                        CPU-offload to `stage2` or `stage3` like this: `stage2 offload` or `stage3 offload`.
+                        Each stage means:
+                            stage1 : optimizer state segmentation
+                            stage2 : optimizer state + gradient segmentation
+                            stage3 : parameter + gradient + optimizer state segmentation
+                            offload ： offload parameters to cpu
+                        NOTICE： stage3 is temporarily unavaliable.
+
+  --sharding_degree
+                        设置sharding的通信组参数，表示通信组的大小。同一个sharding通信组内的参数，进行sharding，分布到不同卡上。
+                        不同sharding通信组之间，相当于单纯的数据并行。此选项只在sharding选项开启时候生效。
+                        默认值为-1，表示所有训练的卡在同一个通信组内。
+                        (`int`, 可选, 默认为 `-1`)
+
+                        Sharding parameter in certain cards group. For example, aussume we use 2 machines each
+                        with 8 cards, then set sharding_degree=8, sharding will only communication inside machine.
+                        default -1 means sharding parameters between all workers. (`int`, *optional*, defaults to `-1`)
+
   --recompute
                         是否使用重计算训练。可以节省显存。
                         重新计算前向过程以获取梯度，减少中间变量显存

examples/language_model/moe/dygraph/framework/group_sharded.py

@@ -35,7 +35,6 @@ from paddle.fluid.framework import in_dygraph_mode
 from paddle.fluid.clip import ClipGradBase, _squared_l2_norm
 from paddle.fluid.dygraph import base as imperative_base
 from paddle.fluid import core, layers, framework
-from paddle.distributed import collective
 from paddle.incubate.distributed.models.moe.grad_clip import ClipGradForMOEByGlobalNorm
 
 # Old version
@@ -72,18 +71,19 @@ class ClipGradForShardedMOEByGlobalNorm(ClipGradForMOEByGlobalNorm):
         global_norm_var_normal, sum_dtype \
             = self.get_l2_norm_pow(normal_params_grads)
         if global_norm_var_normal is not None:
-            collective.all_reduce(global_norm_var_normal,
-                                  op=collective.ReduceOp.SUM,
-                                  group=self.moe_group)
+            paddle.distributed.all_reduce(global_norm_var_normal,
+                                          op=paddle.distributed.ReduceOp.SUM,
+                                          group=self.moe_group)
 
         global_norm_var_moe = None
         if len(moe_params_grads) > 0:
             global_norm_var_moe, _ \
                 = self.get_l2_norm_pow(moe_params_grads, sum_dtype)
             if global_norm_var_moe is not None:
-                collective.all_reduce(global_norm_var_moe,
-                                      op=collective.ReduceOp.SUM,
-                                      group=self.moe_group)
+                paddle.distributed.all_reduce(
+                    global_norm_var_moe,
+                    op=paddle.distributed.ReduceOp.SUM,
+                    group=self.moe_group)
 
         if global_norm_var_normal is None and global_norm_var_moe is None:
             return params_grads

examples/language_model/t5/README.md

@@ -46,12 +46,53 @@ python run_glue.py \
 - `scheduler_type` scheduler类型，可选linear和cosine，默认linear。
 - `output_dir` 表示模型保存路径。
 
+使用trainer进行Fine-tuning:
+```shell
+python -m paddle.distributed.launch --gpus "0,1,2,3" run_glue_trainer.py \
+    --model_name_or_path t5-base \
+    --task_name rte \
+    --max_seq_length 256 \
+    --do_train \
+    --do_eval \
+    --per_device_train_batch_size 16 \
+    --per_device_eval_batch_size 64 \
+    --learning_rate 1e-4 \
+    --weight_decay 0.01 \
+    --warmup_ratio 0.1 \
+    --num_train_epochs 10 \
+    --eval_steps 200 \
+    --logging_steps 20 \
+    --save_steps 200 \
+    --save_total_limit 3 \
+    --metric_for_best_model "eval_accuarcy" \
+    --fp16 false \
+    --fp16_opt_level "O1" \
+    --recompute true \
+    --sharding "stage1" \
+    --overwrite_output_dir \
+    --disable_tqdm true \
+    --output_dir outputs/rte/
+```
+具体参数含义请参见: https://paddlenlp.readthedocs.io/zh/latest/trainer.html
+
 ###### t5-base模型在GLUE开发集上的结果：
 | Model                          | cola  | sst-2  | mrpc        | sts-b             | qqp         | mnli       | qnli | rte   | mean |
 |--------------------------------|-------|-------|-------------|------------------|-------------|-------------|------|-------|-------|
 |                                | mcc   | acc   | acc      | pearson | acc      | acc      | acc  | acc   |         |
 | T5-base-Paddle | 61.74 | 95.18 | 90.44 | 90.09   | 91.60 | 87.18 | 93.56 | 81.95 | 86.4675 |
 
+###### t5_v1_1-base模型在GLUE开发集上的结果：
+使用`run_glue_trainer.py`运行，由于`t5_v1_1-base`没有在glue任务上进行训练过，直接生成label的策略需要的训练时间需要更长。
+| Model                          | cola  | sst-2  | mrpc        | sts-b             | qqp         | mnli       | qnli | rte   |
+|--------------------------------|-------|-------|-------------|------------------|-------------|-------------|------|-------|
+|                                | mcc   | acc   | acc      | pearson | acc      | acc      | acc  | acc   |
+| T5-v1_1-base Paddle | 47.6845 | 94.38 | 84.31 | 87.74   | 88.05 | 85.39 | 90.518 | 65.70 |
+| epoch | 100 | 10 | 100 | 100   | 3 | 3 | 10 | 100 |
+
+注：
+- 直接生成label的finetune方式难度较大，前期基本学习如何正确生成label标签，后期才学习分类任务。
+- 生成的label标签设计，标签差异大一些，效果会更好一些。
+- `qqp`,`mnli`数据集适当增大训练epoch数，可以取得更好效果。
 
 ### GLUE Demo测试
 

examples/language_model/t5/data.py

@@ -50,6 +50,33 @@ GLUE_PROCESSED = collections.OrderedDict([
     ),
 ])
 
+GLUE_1_1_PROCESSED = collections.OrderedDict([
+    ("cola", (["cola sentence: "], ["outrageous", "acceptable"])),
+    ("sst-2", (["sst2 sentence: "], ["negative", "positive"])),
+    (
+        "mrpc",
+        (["mrpc sentence1: ", " sentence2: "], ["nonidentical", "equivalent"]),
+    ),
+    ("sts-b", (["stsb sentence1: ", " sentence2: "], None)),
+    ("qqp", (["qqp question1: ", " question2: "], ["inequable", "duplicate"])),
+    (
+        "mnli",
+        (
+            ["mnli hypothesis: ", " premise: "],
+            ["contradiction", "entailment", "neutral"],
+        ),
+    ),
+    (
+        "qnli",
+        (["qnli question: ", " sentence: "], ["entailment", "contradiction"]),
+    ),
+    (
+        "rte",
+        (["rte sentence1: ",
+          " rte sentence2: "], ["entailment", "contradiction"]),
+    ),
+])
+
 
 def trans_func(example, tokenizer, args):
     task_name = args.task_name

examples/language_model/t5/run_glue_trainer.py

@@ -0,0 +1,429 @@
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import math
+import os
+from functools import partial
+from dataclasses import dataclass, field
+from typing import Optional, Dict, Union, Any, List, Tuple
+
+from tqdm import tqdm
+import paddle
+import paddle.nn as nn
+from paddlenlp.transformers import T5ForConditionalGeneration, T5Tokenizer
+from paddlenlp.datasets import load_dataset
+from paddlenlp.trainer import get_last_checkpoint
+from paddlenlp.data import DataCollatorWithPadding, Pad
+from paddlenlp.trainer import (
+    PdArgumentParser,
+    TrainingArguments,
+    Trainer,
+)
+from paddlenlp.utils.log import logger
+
+from utils import load_pickle, save_pickle, GLUE_METRICS
+from data import GLUE_PROCESSED, GLUE_1_1_PROCESSED
+
+label_length_map = {
+    "cola": 4,
+    "sst-2": 1,
+    "mrpc": 5,
+    "sts-b": 5,
+    "qqp": 5,
+    "mnli": 4,
+    "qnli": 5,
+    "rte": 5,
+}
+
+
+def trans_func(example, tokenizer, args):
+    task_name = args.task_name
+    PROCESSED = GLUE_PROCESSED
+    if "v1_1" in args.cache_dir:
+        PROCESSED = GLUE_1_1_PROCESSED
+    processed, label = PROCESSED[task_name]
+    if label:
+        id2label = dict(zip(range(len(label)), label))
+    else:
+        id2label = None
+
+    is_test = "labels" not in example
+
+    if not is_test:
+        if id2label:
+            label_text = id2label[example["labels"]]
+        else:
+            label_text = str(example["labels"])
+        target = tokenizer(label_text,
+                           return_token_type_ids=False,
+                           return_attention_mask=True)
+
+    if len(processed) == 1:
+        text = processed[0] + example["sentence"]
+    else:
+        text = processed[0] + example["sentence1"] + processed[1] + example[
+            "sentence2"]
+
+    source = tokenizer(
+        text,
+        max_seq_len=args.max_seq_length,
+        padding='max_length',
+        return_token_type_ids=False,
+        return_attention_mask=True,
+    )
+
+    if not is_test:
+        return {
+            "input_ids": source["input_ids"],
+            "attention_mask": source["attention_mask"],
+            "labels": target["input_ids"],
+            "decoder_attention_mask": target["attention_mask"],
+        }
+    else:
+        return {
+            "input_ids": source["input_ids"],
+            "attention_mask": source["attention_mask"]
+        }
+
+
+class BatchDict(object):
+
+    def __init__(self, fn):
+        assert isinstance(fn, (dict)), 'Input pattern not understood. The input of Dict must be a dict with key of input column name and value of collate_fn ' \
+                                   'Received fn=%s' % (str(fn))
+
+        self._fn = fn
+
+        for col_name, ele_fn in self._fn.items():
+            assert callable(
+                ele_fn
+            ), 'Batchify functions must be callable! type(fn[%d]) = %s' % (
+                col_name, str(type(ele_fn)))
+
+    def __call__(self, data):
+
+        ret = {}
+        if len(data) <= 0:
+            return ret
+
+        for col_name, ele_fn in self._fn.items():
+            # skip unused col_name, such as labels in test mode.
+            if col_name not in data[0].keys():
+                continue
+            result = ele_fn([ele[col_name] for ele in data])
+            ret[col_name] = result
+
+        return ret
+
+
+def get_train_dataset(tokenizer, args):
+    filename = os.path.join(args.cache_dir, args.task_name + "_train" + ".pkl")
+
+    if os.path.exists(filename):
+        ds = load_pickle(filename)
+    else:
+        ds = load_dataset("glue", args.task_name, splits="train")
+        ds.map(
+            partial(trans_func, tokenizer=tokenizer, args=args),
+            batched=False,
+            lazy=False,
+        )
+        save_pickle(ds, filename)
+
+    return ds
+
+
+def get_dev_dataset(tokenizer, args):
+    filename = os.path.join(args.cache_dir, args.task_name + "_dev" + ".pkl")
+
+    if os.path.exists(filename):
+        ds = load_pickle(filename)
+    else:
+        ds = load_dataset("glue", args.task_name, splits="dev")
+        ds.map(
+            partial(trans_func, tokenizer=tokenizer, args=args),
+            batched=False,
+            lazy=False,
+        )
+        save_pickle(ds, filename)
+
+    return ds
+
+
+def get_mnli_dev_dataset(tokenizer, args, matched=True):
+    if matched:
+        split = "dev_matched"
+    else:
+        split = "dev_mismatched"
+    filename = os.path.join(args.cache_dir,
+                            args.task_name + f"_{split}" + ".pkl")
+    if os.path.exists(filename):
+        ds = load_pickle(filename)
+    else:
+        ds = load_dataset("glue", args.task_name, splits=split)
+        ds.map(
+            partial(trans_func, tokenizer=tokenizer, args=args),
+            batched=False,
+            lazy=False,
+        )
+        save_pickle(ds, filename)
+
+    return ds
+
+
+@dataclass
+class DataArguments:
+    """
+    Arguments pertaining to what data we are going to input our model for training and eval.
+    Using `PdArgumentParser` we can turn this class
+    into argparse arguments to be able to specify them on
+    the command line.
+    """
+
+    task_name: str = field(
+        default=None,
+        metadata={
+            "help": "The name of the task to use (via the datasets library)."
+        })
+
+    max_seq_length: int = field(
+        default=128,
+        metadata={
+            "help":
+            "The maximum total input sequence length after tokenization. Sequences longer "
+            "than this will be truncated, sequences shorter will be padded."
+        },
+    )
+    cache_dir: str = field(default="./caches",
+                           metadata={"help": "cache dir for datasets."})
+
+
+@dataclass
+class ModelArguments:
+    """
+    Arguments pertaining to which model/config/tokenizer we are going to fine-tune from.
+    """
+
+    model_name_or_path: str = field(
+        default="t5-small",
+        metadata={
+            "help":
+            "Path to pretrained model or model identifier from https://paddlenlp.readthedocs.io/zh/latest/model_zoo/transformers.html"
+        })
+    export_model_dir: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": "Path to directory to store the exported inference model."
+        },
+    )
+
+
+class T5GlueTrainer(Trainer):
+
+    def __init__(self, do_generation: bool, label2id, **kwargs):
+        super().__init__(**kwargs)
+        self.do_generation = do_generation
+        self.label2id = label2id
+
+    def prediction_step(
+        self,
+        model: nn.Layer,
+        inputs: Dict[str, Union[paddle.Tensor, Any]],
+        prediction_loss_only: bool,
+        ignore_keys: Optional[List[str]] = None,
+    ) -> Tuple[Optional[paddle.Tensor], Optional[paddle.Tensor],
+               Optional[paddle.Tensor]]:
+
+        if not self.do_generation:
+            return super().prediction_step(
+                model,
+                inputs,
+                prediction_loss_only=prediction_loss_only,
+                ignore_keys=ignore_keys)
+
+        all_preds = []
+        all_labels = []
+        # source_ids, source_mask, labels, target_mask = batch
+        labels = inputs["labels"]
+        target_mask = inputs["decoder_attention_mask"]
+
+        with paddle.no_grad():
+            outputs = model.generate(
+                input_ids=inputs["input_ids"],
+                attention_mask=inputs["attention_mask"],
+                max_length=5,
+            )[0]
+
+        for p, l, m in zip(outputs.numpy(), labels.numpy(),
+                           target_mask.numpy()):
+            pred = self.tokenizer.decode(p, skip_special_tokens=True).strip()
+            label = self.tokenizer.decode(l[m.astype("bool")],
+                                          skip_special_tokens=True).strip()
+
+            if self.label2id:
+                # for classifaction task.
+                label = self.label2id[label]
+                if pred not in self.label2id:
+                    # set to wrong label if the generated text not in the labal set.
+                    pred = 0
+                    if label == 0:
+                        pred = 1
+                else:
+                    pred = self.label2id[pred]
+            else:
+                # for regression task.
+                label = float(label.replace(" ", ""))
+                try:
+                    pred = float(pred.replace(" ", ""))
+                except Exception as e:
+                    # set to zero if the generated text can not convert to float
+                    pred = 0.0
+
+            all_preds.append(pred)
+            all_labels.append(label)
+
+        all_preds = paddle.to_tensor(all_preds).detach()
+        all_labels = paddle.to_tensor(all_labels).detach()
+
+        return (None, all_preds, all_labels)
+
+
+def main():
+    parser = PdArgumentParser(
+        (ModelArguments, DataArguments, TrainingArguments))
+    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
+
+    if "v1_1" in model_args.model_name_or_path:
+        data_args.cache_dir = "./caches_v1_1"
+    if not os.path.exists(data_args.cache_dir):
+        os.mkdir(data_args.cache_dir)
+
+    # Log model and data config
+    training_args.print_config(model_args, "Model")
+    training_args.print_config(data_args, "Data")
+
+    paddle.set_device(training_args.device)
+
+    # Log on each process the small summary:
+    logger.warning(
+        f"Process rank: {training_args.local_rank}, device: {training_args.device}, world_size: {training_args.world_size}, "
+        +
+        f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16 or training_args.bf16}"
+    )
+
+    # Detecting last checkpoint.
+    last_checkpoint = None
+    if os.path.isdir(
+            training_args.output_dir
+    ) and training_args.do_train and not training_args.overwrite_output_dir:
+        last_checkpoint = get_last_checkpoint(training_args.output_dir)
+        if last_checkpoint is None and len(os.listdir(
+                training_args.output_dir)) > 0:
+            raise ValueError(
+                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
+                "Use --overwrite_output_dir to overcome.")
+        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
+            logger.info(
+                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
+                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
+            )
+
+    PROCESSED = GLUE_PROCESSED
+    if "v1_1" in data_args.cache_dir:
+        PROCESSED = GLUE_1_1_PROCESSED
+    label_name = PROCESSED[data_args.task_name][1]
+    if label_name:
+        label2id = dict(zip(label_name, range(len(label_name))))
+    else:
+        label2id = None
+    metric_list = GLUE_METRICS[data_args.task_name]
+    generate_max_length = label_length_map[data_args.task_name]
+
+    # get model and tokenizer
+    model = T5ForConditionalGeneration.from_pretrained(
+        model_args.model_name_or_path)
+    tokenizer = T5Tokenizer.from_pretrained(model_args.model_name_or_path)
+
+    # get dataloader
+    train_dataset = get_train_dataset(tokenizer, data_args)
+    if data_args.task_name == "mnli":
+        eval_dataset = get_mnli_dev_dataset(tokenizer, data_args, matched=True)
+        eval_dataset_mismatch = get_mnli_dev_dataset(tokenizer,
+                                                     data_args,
+                                                     matched=False)
+    else:
+        eval_dataset = get_dev_dataset(tokenizer, data_args)
+
+    batchify_fn = lambda samples, fn=BatchDict({
+        "input_ids":
+        Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype="int64"),  # input_ids
+        "attention_mask":
+        Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype="int64"
+            ),  # attention_mask
+        "labels":
+        Pad(axis=0, pad_val=-100, dtype="int64"),  # lm_labels
+        "decoder_attention_mask":
+        Pad(axis=0, pad_val=tokenizer.pad_token_id, dtype="int64"
+            ),  # decoder_attention_mask
+    }): fn(samples)
+    data_collator = batchify_fn
+
+    # Define the metrics of tasks.
+    def compute_metrics(p):
+        preds = p.predictions[0] if isinstance(p.predictions,
+                                               tuple) else p.predictions
+
+        results = {}
+        for metric in metric_list:
+            results.update(metric(p.label_ids, preds))
+
+        return results
+
+    trainer = T5GlueTrainer(
+        model=model,
+        criterion=None,
+        args=training_args,
+        data_collator=data_collator,
+        train_dataset=train_dataset if training_args.do_train else None,
+        eval_dataset=eval_dataset if training_args.do_eval else None,
+        tokenizer=tokenizer,
+        compute_metrics=compute_metrics,
+        do_generation=True,
+        label2id=label2id,
+    )
+
+    checkpoint = None
+    if training_args.resume_from_checkpoint is not None:
+        checkpoint = training_args.resume_from_checkpoint
+    elif last_checkpoint is not None:
+        checkpoint = last_checkpoint
+
+    # Training
+    if training_args.do_train:
+        train_result = trainer.train(resume_from_checkpoint=checkpoint)
+        metrics = train_result.metrics
+        # trainer.save_model()
+        trainer.log_metrics("train", metrics)
+        trainer.save_metrics("train", metrics)
+        trainer.save_state()
+
+    # Evaluate and tests model
+    if training_args.do_eval:
+        eval_metrics = trainer.evaluate()
+        trainer.log_metrics("eval", eval_metrics)
+
+
+if __name__ == "__main__":
+    main()