Compare commits
34 Commits
ppdiffuser
...
release/2.
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7c96f95ea6 | ||
|
|
0e8603c21c | ||
|
|
3372a3382a | ||
|
|
63f9d88d9e | ||
|
|
74f46464aa | ||
|
|
f7575220b9 | ||
|
|
e288d7f07c | ||
|
|
37644cc512 | ||
|
|
5c59a81cb7 | ||
|
|
10481e9888 | ||
|
|
097220364b | ||
|
|
9cb33e166b | ||
|
|
dbecb4af2e | ||
|
|
12e2db7422 | ||
|
|
7a6b934547 | ||
|
|
a9eb71b90a | ||
|
|
eeca1a2054 | ||
|
|
196793bb63 | ||
|
|
3427973ee0 | ||
|
|
69be464be8 | ||
|
|
e7a865111b | ||
|
|
83f7c91f71 | ||
|
|
f83cd7394a | ||
|
|
c4932f8e2d | ||
|
|
a4259164bf | ||
|
|
c82e777b30 | ||
|
|
b070e720a0 | ||
|
|
3afa7bd052 | ||
|
|
867407d9cf | ||
|
|
c982271883 | ||
|
|
1bc6c86036 | ||
|
|
34cea03b20 | ||
|
|
4bab6e647d | ||
|
|
45bb58cfcc |
@@ -35,7 +35,7 @@ PaddleNLP是飞桨自然语言处理开发库,具备**易用的文本领域API
|
||||
### 环境依赖
|
||||
|
||||
- python >= 3.6
|
||||
- paddlepaddle >= 2.1
|
||||
- paddlepaddle >= 2.2
|
||||
|
||||
### pip安装
|
||||
|
||||
|
||||
@@ -34,7 +34,7 @@ English | [简体中文](./README.md)
|
||||
### Prerequisites
|
||||
|
||||
* python >= 3.6
|
||||
* paddlepaddle >= 2.1
|
||||
* paddlepaddle >= 2.2
|
||||
|
||||
More information about PaddlePaddle installation please refer to [PaddlePaddle's Website](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/install/conda/linux-conda.html).
|
||||
|
||||
|
||||
BIN
docs/imgs/wechat.png
Normal file
BIN
docs/imgs/wechat.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 388 KiB |
@@ -18,6 +18,12 @@ FasterTokenizer底层为C++实现,同时提供了python接口调用。其可
|
||||
|
||||
以下示例展示FasterERNIE用于文本分类任务。
|
||||
|
||||
## 环境依赖
|
||||
|
||||
* paddlepaddle >= 2.2.1
|
||||
|
||||
* paddlenlp >= 2.2
|
||||
|
||||
## 代码结构说明
|
||||
|
||||
以下是本项目主要代码结构及说明:
|
||||
|
||||
@@ -18,6 +18,12 @@ FasterTokenizer底层为C++实现,同时提供了python接口调用。其可
|
||||
|
||||
以下示例展示FasterERNIE用于序列标注任务。
|
||||
|
||||
## 环境依赖
|
||||
|
||||
* paddlepaddle >= 2.2.1
|
||||
|
||||
* paddlenlp >= 2.2
|
||||
|
||||
## 代码结构说明
|
||||
|
||||
以下是本项目主要代码结构及说明:
|
||||
|
||||
@@ -1,9 +1,8 @@
|
||||
# Faster Tokenizer 性能测试
|
||||
# 飞桨FasterTokenizer性能测试
|
||||
|
||||
为了进一步对比Faster Tokenizer的性能,我们选取的业界对于Transformer类常用的Tokenizer分词工具进行对比。
|
||||
我们以 bert-base-chinese 模型为例,对比的Tokenizer分词工具有以下选择:
|
||||
在PaddleNLP v2.2.0版本中PaddleNLP推出了高性能的Transformer类文本分词器,简称飞桨FasterTokenizer。为了验证飞桨FasterTokenizer的性能快的特点,PaddleNLP选取了业内常见的一些文本分词器进行了性能对比比较,主要进行性能参考的是HuggingFace BertTokenizer, Tensorflow-text BertTokenizer. 我们以 bert-base-chinese 模型为例进行了文本分词性能实验对比,在中文的数据下进行性能对比实验,下面是具体实验设置信息:
|
||||
|
||||
* HuggingFace BertTokenizer: 以下简称 HFTokenizer。
|
||||
* [HuggingFace Tokenizers(Python)](https://github.com/huggingface/tokenizers):
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer
|
||||
@@ -11,7 +10,7 @@ from transformers import AutoTokenizer
|
||||
hf_tokenizer = AutoTokenizer.from_pretrained("bert-base-chinese", use_fast=False)
|
||||
```
|
||||
|
||||
* [HuggingFace BertTokenizerFast](https://github.com/huggingface/tokenizers): 以下简称 HFFastTokenizer。
|
||||
* [HuggingFace Tokenizers(Rust)](https://github.com/huggingface/tokenizers):
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer
|
||||
@@ -19,7 +18,7 @@ from transformers import AutoTokenizer
|
||||
hf_tokenizer_fast = AutoTokenizer.from_pretrained("bert-base-chinese", use_fast=True)
|
||||
```
|
||||
|
||||
* [TensorFlow-Text BertTokenizer](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer):以下简称 TFTokenizer。
|
||||
* [TensorFlow-Text](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer):
|
||||
|
||||
```python
|
||||
import tensorflow_text as tf_text
|
||||
@@ -28,22 +27,27 @@ import tensorflow_text as tf_text
|
||||
tf_tokenizer = tf_text.BertTokenizer(vocab)
|
||||
```
|
||||
|
||||
* PaddleNLP BertTokenizer:以下简称 PPNLPTokenizer
|
||||
* [飞桨FasterTokenizer](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/paddlenlp/experimental):
|
||||
|
||||
```python
|
||||
from paddlenlp.transformers import BertTokenizer
|
||||
from paddlenlp.experimental import FasterTokenizer
|
||||
|
||||
faster_tokenizer = FasterTokenizer.from_pretrained("bert-base-chinese")
|
||||
|
||||
py_tokenizer = BertTokenizer.from_pretrained("bert-base-chinese")
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 环境依赖
|
||||
|
||||
* paddlenlp >= 2.2.0
|
||||
* paddlepaddle >= 2.2.1
|
||||
|
||||
* paddlenlp >= 2.2
|
||||
|
||||
* transformers == 4.11.3
|
||||
|
||||
* tokenizers == 0.10.3
|
||||
|
||||
* tensorflow_text == 2.5.0
|
||||
|
||||
|
||||
@@ -64,10 +68,6 @@ python perf.py
|
||||
|
||||
- 测试结果:
|
||||
|
||||
文本序列长度为128, 线程数为16,batch_size=32的性能对比结果:
|
||||
<center><img width="1343" alt="图片" src="https://user-images.githubusercontent.com/16698950/145664356-0b766d5a-9ff1-455a-bb85-1ee51e2ad77d.png"></center>
|
||||
|
||||
|
||||
<center><img src="https://ai-studio-static-online.cdn.bcebos.com/9d46bfe903614444b4cf9e63206b28ee06f06c5d5cb04da58bb206431904af00" ></center>
|
||||
<br> <center> Tokenizer性能对比 </center></br>
|
||||
|
||||
从以上结果可以看出,FasterTokenizer性能远远超过了其他Tokenizer, 高达HFFastTokenizer性能20倍。
|
||||
飞桨FasterTokenizer与其他框架性能的对比,是在固定文本长度在不同batch size下的分词吞吐量。纵坐标是对数坐标,单位是1w tokens/秒。随着batch size的增大,飞桨FasterTokenizer速度会远远超过其他同类产品的实现,尤其是在大batch文本上飞桨框架能充分发挥多核机器的优势,取得领先的效果。
|
||||
|
||||
Reference in New Issue
Block a user