Compare commits

...

34 Commits

Author SHA1 Message Date
chenxiaozeng
7c96f95ea6 Add files via upload 2022-05-13 00:01:38 +08:00
chenxiaozeng
0e8603c21c Delete wechat.png 2022-05-13 00:01:15 +08:00
chenxiaozeng
3372a3382a Add files via upload 2022-05-04 22:14:49 +08:00
chenxiaozeng
63f9d88d9e Delete wechat.png 2022-05-04 22:14:29 +08:00
chenxiaozeng
74f46464aa Add files via upload 2022-04-26 15:27:46 +08:00
chenxiaozeng
f7575220b9 Delete wechat.png 2022-04-26 15:26:40 +08:00
chenxiaozeng
e288d7f07c Add files via upload 2022-04-22 16:24:56 +08:00
chenxiaozeng
37644cc512 Delete wechat.png 2022-04-22 16:24:30 +08:00
chenxiaozeng
5c59a81cb7 Add files via upload 2022-04-15 17:41:12 +08:00
chenxiaozeng
10481e9888 Delete wechat.png 2022-04-15 17:40:51 +08:00
chenxiaozeng
097220364b Add files via upload 2022-04-15 17:38:50 +08:00
chenxiaozeng
9cb33e166b Delete wechat.png 2022-04-15 17:38:27 +08:00
chenxiaozeng
dbecb4af2e Add files via upload 2022-04-15 17:35:01 +08:00
chenxiaozeng
12e2db7422 Delete wechat.png 2022-04-15 17:34:25 +08:00
chenxiaozeng
7a6b934547 Delete wechat.png 2022-04-08 22:39:21 +08:00
chenxiaozeng
a9eb71b90a Add files via upload 2022-04-08 22:33:45 +08:00
chenxiaozeng
eeca1a2054 Delete wechat.png 2022-04-08 22:33:04 +08:00
chenxiaozeng
196793bb63 Add files via upload 2022-04-08 22:30:01 +08:00
chenxiaozeng
3427973ee0 Delete wechat.png 2022-04-08 22:29:41 +08:00
chenxiaozeng
69be464be8 Add files via upload 2022-04-08 22:28:30 +08:00
chenxiaozeng
e7a865111b Delete wechat.png 2022-04-08 22:28:09 +08:00
chenxiaozeng
83f7c91f71 Add files via upload 2022-04-01 20:46:09 +08:00
chenxiaozeng
f83cd7394a Delete wechat.png 2022-04-01 20:45:52 +08:00
chenxiaozeng
c4932f8e2d Update group code
Update group code
2022-03-25 22:52:34 +08:00
chenxiaozeng
a4259164bf Update group code 2022-03-25 22:51:07 +08:00
chenxiaozeng
c82e777b30 Delete wechat.png 2022-03-25 22:40:33 +08:00
chenxiaozeng
b070e720a0 Update group code
Update group code
2022-03-25 22:37:47 +08:00
chenxiaozeng
3afa7bd052 add wechat group code
add wechat group code
2022-03-18 20:34:11 +08:00
wawltor
867407d9cf Add the tokeniers version in fastertokenizer (#1462) 2021-12-14 15:38:51 +08:00
wawltor
c982271883 Merge pull request #1447 from Steffy-zxf/up-2.2
[cherrypick-2.2] update paddle verion requires
2021-12-11 14:20:20 +08:00
Steffy-zxf
1bc6c86036 update paddle verion requires 2021-12-11 14:08:47 +08:00
wawltor
34cea03b20 Update README.md 2021-12-11 13:20:08 +08:00
wawltor
4bab6e647d Merge pull request #1446 from wawltor/faster_tokenizer_cp_new
update the document in the FasterTokenizer examples
2021-12-11 13:11:38 +08:00
wawltor
45bb58cfcc Update README.md 2021-12-11 05:09:12 +00:00
6 changed files with 30 additions and 18 deletions

View File

@@ -35,7 +35,7 @@ PaddleNLP是飞桨自然语言处理开发库,具备**易用的文本领域API
### 环境依赖
- python >= 3.6
- paddlepaddle >= 2.1
- paddlepaddle >= 2.2
### pip安装

View File

@@ -34,7 +34,7 @@ English | [简体中文](./README.md)
### Prerequisites
* python >= 3.6
* paddlepaddle >= 2.1
* paddlepaddle >= 2.2
More information about PaddlePaddle installation please refer to [PaddlePaddle's Website](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/install/conda/linux-conda.html).

BIN
docs/imgs/wechat.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 388 KiB

View File

@@ -18,6 +18,12 @@ FasterTokenizer底层为C++实现,同时提供了python接口调用。其可
以下示例展示FasterERNIE用于文本分类任务。
## 环境依赖
* paddlepaddle >= 2.2.1
* paddlenlp >= 2.2
## 代码结构说明
以下是本项目主要代码结构及说明:

View File

@@ -18,6 +18,12 @@ FasterTokenizer底层为C++实现,同时提供了python接口调用。其可
以下示例展示FasterERNIE用于序列标注任务。
## 环境依赖
* paddlepaddle >= 2.2.1
* paddlenlp >= 2.2
## 代码结构说明
以下是本项目主要代码结构及说明:

View File

@@ -1,9 +1,8 @@
# Faster Tokenizer 性能测试
# 飞桨FasterTokenizer性能测试
为了进一步对比Faster Tokenizer的性能,我们选取的业界对于Transformer类常用的Tokenizer分词工具进行对比。
我们以 bert-base-chinese 模型为例,对比的Tokenizer分词工具有以下选择:
在PaddleNLP v2.2.0版本中PaddleNLP推出了高性能的Transformer类文本分词器,简称飞桨FasterTokenizer。为了验证飞桨FasterTokenizer的性能快的特点,PaddleNLP选取了业内常见的一些文本分词器进行了性能对比比较,主要进行性能参考的是HuggingFace BertTokenizer, Tensorflow-text BertTokenizer. 我们以 bert-base-chinese 模型为例进行了文本分词性能实验对比,在中文的数据下进行性能对比实验,下面是具体实验设置信息:
* HuggingFace BertTokenizer: 以下简称 HFTokenizer。
* [HuggingFace Tokenizers(Python)](https://github.com/huggingface/tokenizers):
```python
from transformers import AutoTokenizer
@@ -11,7 +10,7 @@ from transformers import AutoTokenizer
hf_tokenizer = AutoTokenizer.from_pretrained("bert-base-chinese", use_fast=False)
```
* [HuggingFace BertTokenizerFast](https://github.com/huggingface/tokenizers): 以下简称 HFFastTokenizer。
* [HuggingFace Tokenizers(Rust)](https://github.com/huggingface/tokenizers):
```python
from transformers import AutoTokenizer
@@ -19,7 +18,7 @@ from transformers import AutoTokenizer
hf_tokenizer_fast = AutoTokenizer.from_pretrained("bert-base-chinese", use_fast=True)
```
* [TensorFlow-Text BertTokenizer](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer):以下简称 TFTokenizer。
* [TensorFlow-Text](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer):
```python
import tensorflow_text as tf_text
@@ -28,22 +27,27 @@ import tensorflow_text as tf_text
tf_tokenizer = tf_text.BertTokenizer(vocab)
```
* PaddleNLP BertTokenizer:以下简称 PPNLPTokenizer
* [飞桨FasterTokenizer](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/paddlenlp/experimental):
```python
from paddlenlp.transformers import BertTokenizer
from paddlenlp.experimental import FasterTokenizer
faster_tokenizer = FasterTokenizer.from_pretrained("bert-base-chinese")
py_tokenizer = BertTokenizer.from_pretrained("bert-base-chinese")
```
## 环境依赖
* paddlenlp >= 2.2.0
* paddlepaddle >= 2.2.1
* paddlenlp >= 2.2
* transformers == 4.11.3
* tokenizers == 0.10.3
* tensorflow_text == 2.5.0
@@ -64,10 +68,6 @@ python perf.py
- 测试结果:
文本序列长度为128, 线程数为16,batch_size=32的性能对比结果:
<center><img width="1343" alt="图片" src="https://user-images.githubusercontent.com/16698950/145664356-0b766d5a-9ff1-455a-bb85-1ee51e2ad77d.png"></center>
<center><img src="https://ai-studio-static-online.cdn.bcebos.com/9d46bfe903614444b4cf9e63206b28ee06f06c5d5cb04da58bb206431904af00" ></center>
<br> <center> Tokenizer性能对比 </center></br>
从以上结果可以看出,FasterTokenizer性能远远超过了其他Tokenizer, 高达HFFastTokenizer性能20倍。
飞桨FasterTokenizer与其他框架性能的对比,是在固定文本长度在不同batch size下的分词吞吐量。纵坐标是对数坐标,单位是1w tokens/秒。随着batch size的增大,飞桨FasterTokenizer速度会远远超过其他同类产品的实现,尤其是在大batch文本上飞桨框架能充分发挥多核机器的优势,取得领先的效果。