Kimirock/PaddleNLP
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[LLM] Add DeepseekV3 (#9738)

Browse Source 
* update deepseek-v2

* add deepseek_v3

* update for deepseekv3

* update deepseekv3 model_ids
This commit is contained in:
Weiguo Zhu
2025-01-10 14:15:22 +08:00
committed by GitHub
parent cf5e3e72e1
commit 2c556e76fb
22 changed files with 573 additions and 285 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
README.md
View File

@@ -90,6 +90,7 @@
| [ChatGLM2](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/llm/config/chatglm2) | THUDM/chatglm2-6b |
| [ChatGLM3](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/llm/config/chatglm2) | THUDM/chatglm3-6b |
| [DeepSeekV2](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/llm/config/deepseek-v2) | deepseek-ai/DeepSeek-V2, deepseek-ai/DeepSeek-V2-Chat, deepseek-ai/DeepSeek-V2-Lite, deepseek-ai/DeepSeek-V2-Lite-Chat, deepseek-ai/DeepSeek-Coder-V2-Base, deepseek-ai/DeepSeek-Coder-V2-Instruct, deepseek-ai/DeepSeek-Coder-V2-Lite-Base, deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct |
| [DeepSeekV3](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/llm/config/deepseek-v2) | deepseek-ai/DeepSeek-V3, deepseek-ai/DeepSeek-V3-Base |
| [Gemma](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/llm/config/gemma) | google/gemma-7b, google/gemma-7b-it, google/gemma-2b, google/gemma-2b-it |
| [Mistral](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/llm/config/mistral) | mistralai/Mistral-7B-Instruct-v0.3, mistralai/Mistral-7B-v0.1 |
| [Mixtral](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/llm/config/mixtral) | mistralai/Mixtral-8x7B-Instruct-v0.1 |
@@ -130,19 +131,19 @@
| Model | Pretrain | SFT | LoRA | FlashMask | Prefix Tuning | DPO/SimPO/ORPO/KTO | RLHF | Mergekit | Quantization |
|--------------------------------------------|:--------:|:---:|:----:|:---------:|:-------------:|:--------------:|:----:|:-----:|:------------:|
| [Llama](./llm/config/llama) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Qwen](./llm/config/qwen) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 🚧 | ✅ | 🚧 |
| [Mixtral](./llm/config/mixtral) | ✅ | ✅ | ✅ | 🚧 | 🚧 | ✅ | 🚧 | ✅ | 🚧 |
| [Mistral](./llm/config/mistral) | ✅ | ✅ | ✅ | 🚧 | ✅ | ✅ | 🚧 | ✅ | 🚧 |
| [Baichuan/Baichuan2](./llm/config/llama) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 🚧 | ✅ | ✅ |
| [ChatGLM-6B](./llm/config/chatglm) | ✅ | ✅ | ✅ | 🚧 | ✅ | 🚧 | 🚧 | ✅ | ✅ |
| [ChatGLM2/ChatGLM3](./llm/config/chatglm2) | ✅ | ✅ | ✅ | 🚧 | ✅ | ✅ | 🚧 | ✅ | ✅ |
| [Bloom](./llm/config/bloom) | ✅ | ✅ | ✅ | 🚧 | ✅ | 🚧 | 🚧 | ✅ | ✅ |
| [GPT-3](./llm/config/gpt-3) | ✅ | ✅ | 🚧 | 🚧 | 🚧 | 🚧 | 🚧 | ✅ | 🚧 |
| [OPT](./llm/config/opt) | ✅ | ✅ | ✅ | 🚧 | 🚧 | 🚧 | 🚧 | ✅ | 🚧 |
| [Gemma](./llm/config/gemma) | ✅ | ✅ | ✅ | 🚧 | 🚧 | ✅ | 🚧 | ✅ | 🚧 |
| [Yuan](./llm/config/yuan) | ✅ | ✅ | ✅ | 🚧 | 🚧 | ✅ | 🚧 | ✅ | 🚧 |
|--------------------------------------------|:--------:|:---:|:----:|:---------:|:-------------:|:------------------:|:----:|:--------:|:------------:|
| [Llama](./llm/config/llama) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Qwen](./llm/config/qwen) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 🚧 | ✅ | 🚧 |
| [Mixtral](./llm/config/mixtral) | ✅ | ✅ | ✅ | 🚧 | 🚧 | ✅ | 🚧 | ✅ | 🚧 |
| [Mistral](./llm/config/mistral) | ✅ | ✅ | ✅ | 🚧 | ✅ | ✅ | 🚧 | ✅ | 🚧 |
| [Baichuan/Baichuan2](./llm/config/llama) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 🚧 | ✅ | ✅ |
| [ChatGLM-6B](./llm/config/chatglm) | ✅ | ✅ | ✅ | 🚧 | ✅ | 🚧 | 🚧 | ✅ | ✅ |
| [ChatGLM2/ChatGLM3](./llm/config/chatglm2) | ✅ | ✅ | ✅ | 🚧 | ✅ | ✅ | 🚧 | ✅ | ✅ |
| [Bloom](./llm/config/bloom) | ✅ | ✅ | ✅ | 🚧 | ✅ | 🚧 | 🚧 | ✅ | ✅ |
| [GPT-3](./llm/config/gpt-3) | ✅ | ✅ | 🚧 | 🚧 | 🚧 | 🚧 | 🚧 | ✅ | 🚧 |
| [OPT](./llm/config/opt) | ✅ | ✅ | ✅ | 🚧 | 🚧 | 🚧 | 🚧 | ✅ | 🚧 |
| [Gemma](./llm/config/gemma) | ✅ | ✅ | ✅ | 🚧 | 🚧 | ✅ | 🚧 | ✅ | 🚧 |
| [Yuan](./llm/config/yuan) | ✅ | ✅ | ✅ | 🚧 | 🚧 | ✅ | 🚧 | ✅ | 🚧 |
* [大模型推理](./llm/docs/predict/inference.md)已支持 LLaMA 系列、Qwen 系列、Mistral 系列、ChatGLM 系列、Bloom 系列和 Baichuan 系列，支持 Weight Only INT8及 INT4推理，支持 WAC（权重、激活、Cache KV）进行 INT8、FP8量化的推理，【LLM】模型推理支持列表如下：
| 模型名称/量化类型支持 | FP16/BF16 | WINT8 | WINT4 | INT8-A8W8 | FP8-A8W8 | INT8-A8W8C8 |

2
paddlenlp/generation/utils.py
View File

@@ -742,7 +742,7 @@ class GenerationMixin(object):
# ['是的', '嗯嗯']
"""
if generation_config is None:
if self.generation_config._from_model_config:
if self.generation_config is None or self.generation_config._from_model_config:
new_generation_config = GenerationConfig.from_model_config(self.config)
if new_generation_config != self.generation_config:
logger.warning(

1
paddlenlp/quantization/quantization_config.py
View File

@@ -52,6 +52,7 @@ class QuantizationConfig:
weight_double_quant_block_size=256,
weight_quant_method="abs_max_channel_wise",
act_quant_method="abs_max",
**kwargs,
):
if weight_quantize_algo is not None and weight_quantize_algo not in [
"weight_only_int8",

324
paddlenlp/transformers/__init__.py
View File

@@ -56,102 +56,185 @@ from .bert.tokenizer import *
from .bert.configuration import *
# isort: split
from .gpt import *
from .roberta.modeling import *
from .roberta.tokenizer import *
from .roberta.configuration import *
from .electra.modeling import *
from .electra.tokenizer import *
from .electra.configuration import *
from .albert.configuration import *
from .albert.modeling import *
from .albert.tokenizer import *
from .bit.modeling import *
from .bit.configuration import *
from .bit.image_processing import *
from .artist.configuration import *
from .artist.modeling import *
from .artist.tokenizer import *
from .auto.configuration import *
from .auto.image_processing import *
from .auto.modeling import *
from .auto.processing import *
from .auto.tokenizer import *
from .bart.configuration import *
from .bart.modeling import *
from .bart.tokenizer import *
from .bart.configuration import *
from .bert_japanese.tokenizer import *
from .bigbird.modeling import *
from .bigbird.configuration import *
from .bigbird.modeling import *
from .bigbird.tokenizer import *
from .bit.configuration import *
from .bit.image_processing import *
from .bit.modeling import *
from .blenderbot.configuration import *
from .blenderbot.modeling import *
from .blenderbot.tokenizer import *
from .blenderbot.configuration import *
from .blenderbot_small.configuration import *
from .blenderbot_small.modeling import *
from .blenderbot_small.tokenizer import *
from .blenderbot_small.configuration import *
from .blip.configuration import *
from .blip.image_processing import *
from .blip.modeling import *
from .blip.modeling_text import *
from .blip.configuration import *
from .blip.processing import *
from .blip.image_processing import *
from .blip_2.configuration import *
from .blip_2.modeling import *
from .blip_2.processing import *
from .bloom.configuration import *
from .bloom.modeling import *
from .bloom.tokenizer import *
from .bloom.tokenizer_fast import *
from .chatglm.configuration import *
from .chatglm.modeling import *
from .chatglm.tokenizer import *
from .chatglm_v2.configuration import *
from .chatglm_v2.modeling import *
from .chatglm_v2.modeling_pp import *
from .chatglm_v2.tokenizer import *
from .chinesebert.configuration import *
from .chinesebert.modeling import *
from .chinesebert.tokenizer import *
from .chineseclip.configuration import *
from .chineseclip.feature_extraction import *
from .chineseclip.image_processing import *
from .chineseclip.modeling import *
from .chineseclip.processing import *
from .chineseclip.tokenizer import *
from .clap.configuration import *
from .clap.feature_extraction import *
from .clap.modeling import *
from .clap.processing import *
from .clip.configuration import *
from .clip.feature_extraction import *
from .clip.image_processing import *
from .clip.modeling import *
from .clip.processing import *
from .clip.tokenizer import *
from .clipseg.configuration import *
from .clipseg.image_processing import *
from .clipseg.modeling import *
from .clipseg.processing import *
from .codegen.configuration import *
from .codegen.modeling import *
from .codegen.tokenizer import *
from .convbert.configuration import *
from .convbert.modeling import *
from .convbert.tokenizer import *
from .ctrl.configuration import *
from .ctrl.modeling import *
from .ctrl.tokenizer import *
from .ctrl.configuration import *
from .dallebart.configuration import *
from .dallebart.modeling import *
from .dallebart.tokenizer import *
from .deberta.configuration import *
from .deberta.modeling import *
from .deberta.tokenizer import *
from .deberta_v2.configuration import *
from .deberta_v2.modeling import *
from .deberta_v2.tokenizer import *
from .deepseek_v2.configuration import *
from .deepseek_v2.modeling import *
from .deepseek_v2.tokenizer_fast import *
from .deepseek_v2.configuration import *
from .dpt.modeling import *
from .dpt.configuration import *
from .dpt.image_processing import *
from .deepseek_v3.configuration import *
from .deepseek_v3.modeling import *
from .distilbert.configuration import *
from .distilbert.modeling import *
from .distilbert.tokenizer import *
from .dpt.configuration import *
from .dpt.image_processing import *
from .dpt.modeling import *
from .electra.configuration import *
from .electra.modeling import *
from .electra.tokenizer import *
from .ernie.configuration import *
from .ernie.modeling import *
from .ernie.tokenizer import *
from .ernie_code.configuration import *
from .ernie_code.modeling import *
from .ernie_code.tokenizer import *
from .ernie_ctm.configuration import *
from .ernie_ctm.modeling import *
from .ernie_ctm.tokenizer import *
from .ernie_ctm.configuration import *
from .ernie_doc.configuration import *
from .ernie_doc.modeling import *
from .ernie_doc.tokenizer import *
from .ernie_doc.configuration import *
from .ernie_gen.modeling import ErnieForGeneration
from .ernie_gram.configuration import *
from .ernie_gram.modeling import *
from .ernie_gram.tokenizer import *
from .ernie_gram.configuration import *
from .ernie_layout.configuration import *
from .ernie_layout.modeling import *
from .ernie_layout.tokenizer import *
from .ernie_layout.configuration import *
from .ernie_m.configuration import *
from .ernie_m.modeling import *
from .ernie_m.tokenizer import *
from .ernie_vil.configuration import *
from .ernie_vil.feature_extraction import *
from .ernie_vil.image_processing import *
from .ernie_vil.modeling import *
from .ernie_vil.processing import *
from .ernie_vil.tokenizer import *
from .fnet.configuration import *
from .fnet.modeling import *
from .fnet.tokenizer import *
from .fnet.configuration import *
from .funnel.configuration import *
from .funnel.modeling import *
from .funnel.tokenizer import *
from .funnel.configuration import *
from .llama import *
from .gau_alpha.configuration import *
from .gau_alpha.modeling import *
from .gau_alpha.tokenizer import *
from .gemma import *
from .glm.configuration import *
from .glm.modeling import *
from .glm.tokenizer import *
from .gpt import *
from .gptj.configuration import *
from .gptj.modeling import *
from .gptj.tokenizer import *
from .jamba.configuration import *
from .jamba.modeling import *
from .jamba.tokenizer import *
from .layoutlm.configuration import *
from .layoutlm.modeling import *
from .layoutlm.tokenizer import *
from .layoutlmv2.configuration import *
from .layoutlmv2.modeling import *
from .layoutlmv2.tokenizer import *
from .layoutlmv2.configuration import *
from .layoutxlm.configuration import *
from .layoutxlm.modeling import *
from .layoutxlm.tokenizer import *
from .layoutxlm.configuration import *
from .llama import *
from .luke.configuration import *
from .luke.modeling import *
from .luke.tokenizer import *
from .luke.configuration import *
from .mamba.configuration import *
from .mamba.modeling import *
from .mamba.tokenizer import *
from .mbart.configuration import *
from .mbart.modeling import *
from .mbart.tokenizer import *
from .mbart.configuration import *
from .megatronbert.configuration import *
from .megatronbert.modeling import *
from .megatronbert.tokenizer import *
from .megatronbert.configuration import *
from .prophetnet.modeling import *
from .prophetnet.tokenizer import *
from .prophetnet.configuration import *
from .minigpt4.configuration import *
from .minigpt4.image_processing import *
from .minigpt4.modeling import *
from .minigpt4.processing import *
from .mistral.configuration import *
from .mistral.modeling import *
from .mixtral.configuration import *
from .mixtral.modeling import *
from .mobilebert.configuration import *
from .mobilebert.modeling import *
from .mobilebert.tokenizer import *
@@ -163,158 +246,77 @@ from .mt5.modeling import *
from .nezha.configuration import *
from .nezha.modeling import *
from .nezha.tokenizer import *
from .nystromformer.configuration import *
from .nystromformer.modeling import *
from .nystromformer.tokenizer import *
from .opt.configuration import *
from .opt.modeling import *
from .optimization import *
from .pegasus.configuration import *
from .pegasus.modeling import *
from .pegasus.tokenizer import *
from .ppminilm.modeling import *
from .ppminilm.tokenizer import *
from .prophetnet.configuration import *
from .prophetnet.modeling import *
from .prophetnet.tokenizer import *
from .qwen import *
from .qwen2 import *
from .qwen2_moe import *
from .reformer.configuration import *
from .reformer.modeling import *
from .reformer.tokenizer import *
from .reformer.configuration import *
from .rembert.configuration import *
from .rembert.modeling import *
from .rembert.tokenizer import *
from .rembert.configuration import *
from .roformer.modeling import *
from .roberta.configuration import *
from .roberta.modeling import *
from .roberta.tokenizer import *
from .roformer.configuration import *
from .roformer.modeling import *
from .roformer.tokenizer import *
from .roformerv2.configuration import *
from .roformerv2.modeling import *
from .roformerv2.tokenizer import *
from .rw.configuration import *
from .rw.modeling import *
from .rw.tokenizer import *
from .semantic_search.modeling import *
from .skep.configuration import *
from .skep.modeling import *
from .skep.tokenizer import *
from .speecht5.configuration import *
from .speecht5.feature_extraction import *
from .speecht5.modeling import *
from .speecht5.processing import *
from .speecht5.tokenizer import *
from .squeezebert.configuration import *
from .squeezebert.modeling import *
from .squeezebert.tokenizer import *
from .squeezebert.configuration import *
from .t5.configuration import *
from .t5.modeling import *
from .t5.tokenizer import *
from .t5.configuration import *
from .tinybert.configuration import *
from .tinybert.modeling import *
from .tinybert.tokenizer import *
from .transformer.modeling import *
from .unified_transformer.configuration import *
from .unified_transformer.modeling import *
from .unified_transformer.tokenizer import *
from .unified_transformer.configuration import *
from .ernie_code.tokenizer import *
from .ernie_code.modeling import *
from .ernie_code.configuration import *
from .ernie_vil.configuration import *
from .ernie_vil.modeling import *
from .ernie_vil.feature_extraction import *
from .ernie_vil.tokenizer import *
from .ernie_vil.processing import *
from .ernie_vil.image_processing import *
from .unimo.configuration import *
from .unimo.modeling import *
from .unimo.tokenizer import *
from .unimo.configuration import *
from .xlnet.modeling import *
from .xlnet.tokenizer import *
from .xlnet.configuration import *
from .visualglm.configuration import *
from .visualglm.image_processing import *
from .visualglm.modeling import *
from .visualglm.processing import *
from .xlm.configuration import *
from .xlm.modeling import *
from .xlm.tokenizer import *
from .xlm.configuration import *
from .xlnet.configuration import *
from .xlnet.modeling import *
from .xlnet.tokenizer import *
from .xlm_roberta.modeling import *
from .xlm_roberta.tokenizer import *
from .xlm_roberta.configuration import *
from .gau_alpha.modeling import *
from .gau_alpha.tokenizer import *
from .gau_alpha.configuration import *
from .gemma import *
from .roformerv2.modeling import *
from .roformerv2.tokenizer import *
from .roformerv2.configuration import *
from .optimization import *
from .opt.configuration import *
from .opt.modeling import *
from .auto.modeling import *
from .auto.tokenizer import *
from .auto.processing import *
from .auto.image_processing import *
from .auto.configuration import *
from .codegen.modeling import *
from .codegen.tokenizer import *
from .codegen.configuration import *
from .artist.modeling import *
from .artist.tokenizer import *
from .artist.configuration import *
from .dallebart.modeling import *
from .dallebart.tokenizer import *
from .dallebart.configuration import *
from .clip.modeling import *
from .clip.configuration import *
from .clip.feature_extraction import *
from .clip.tokenizer import *
from .clip.processing import *
from .clip.image_processing import *
from .chineseclip.modeling import *
from .chineseclip.configuration import *
from .chineseclip.feature_extraction import *
from .chineseclip.processing import *
from .chineseclip.image_processing import *
from .chineseclip.tokenizer import *
from .gptj.modeling import *
from .gptj.tokenizer import *
from .gptj.configuration import *
from .pegasus.modeling import *
from .pegasus.tokenizer import *
from .pegasus.configuration import *
from .glm.configuration import *
from .glm.modeling import *
from .glm.tokenizer import *
from .nystromformer.configuration import *
from .nystromformer.modeling import *
from .nystromformer.tokenizer import *
from .bloom.configuration import *
from .bloom.modeling import *
from .bloom.tokenizer import *
from .bloom.tokenizer_fast import *
from .clipseg.configuration import *
from .clipseg.modeling import *
from .clipseg.processing import *
from .clipseg.image_processing import *
from .blip_2.modeling import *
from .blip_2.configuration import *
from .blip_2.processing import *
from .chatglm.configuration import *
from .chatglm.modeling import *
from .chatglm.tokenizer import *
from .chatglm_v2.configuration import *
from .chatglm_v2.modeling import *
from .chatglm_v2.modeling_pp import *
from .chatglm_v2.tokenizer import *
from .speecht5.configuration import *
from .speecht5.modeling import *
from .speecht5.tokenizer import *
from .speecht5.processing import *
from .speecht5.feature_extraction import *
from .minigpt4.modeling import *
from .minigpt4.configuration import *
from .minigpt4.processing import *
from .minigpt4.image_processing import *
from .clap.configuration import *
from .clap.feature_extraction import *
from .clap.modeling import *
from .clap.processing import *
from .visualglm.modeling import *
from .visualglm.configuration import *
from .visualglm.processing import *
from .visualglm.image_processing import *
from .rw.modeling import *
from .rw.configuration import *
from .rw.tokenizer import *
from .mistral.modeling import *
from .mistral.configuration import *
from .qwen import *
from .mixtral.modeling import *
from .mixtral.configuration import *
from .deberta.modeling import *
from .deberta.tokenizer import *
from .deberta.configuration import *
from .deberta_v2.modeling import *
from .deberta_v2.tokenizer import *
from .deberta_v2.configuration import *
from .qwen2 import *
from .qwen2_moe import *
from .yuan import *
from .mamba.configuration import *
from .mamba.modeling import *
from .mamba.tokenizer import *
from .jamba.modeling import *
from .jamba.configuration import *
from .jamba.tokenizer import *

2
paddlenlp/transformers/artist/configuration.py
View File

@@ -14,7 +14,7 @@
""" MBart model configuration"""
from __future__ import annotations
from paddlenlp.transformers import GPTConfig
from ..gpt.configuration import GPTConfig
__all__ = ["ARTIST_PRETRAINED_INIT_CONFIGURATION", "ARTIST_PRETRAINED_RESOURCE_FILES_MAP", "ArtistConfig"]

4
paddlenlp/transformers/auto/configuration.py
View File

@@ -56,6 +56,8 @@ CONFIG_MAPPING_NAMES = OrderedDict(
("dallebart", "DalleBartConfig"),
("deberta", "DebertaConfig"),
("debertav2", "DebertaV2Config"),
("deepseek_v2", "DeepseekV2Config"),
("deepseek_v3", "DeepseekV3Config"),
("distilbert", "DistilBertConfig"),
("dpt", "DPTConfig"),
("electra", "ElectraConfig"),
@@ -146,6 +148,8 @@ MODEL_NAMES_MAPPING = OrderedDict(
("dallebart", "DalleBart"),
("deberta", "Deberta"),
("debertav2", "DebertaV2"),
("deepseek_v2", "DeepseekV2"),
("deepseek_v3", "DeepseekV3"),
("distilbert", "DistilBert"),
("dpt", "DPT"),
("electra", "Electra"),

2
paddlenlp/transformers/auto/modeling.py
View File

@@ -57,6 +57,8 @@ MAPPING_NAMES = OrderedDict(
("CTRL", "ctrl"),
("DistilBert", "distilbert"),
("DalleBart", "dallebart"),
("DeepseekV2", "deepseek_v2"),
("DeepseekV3", "deepseek_v3"),
("Electra", "electra"),
("ErnieViL", "ernie_vil"),
("ErnieCtm", "ernie_ctm"),

53
paddlenlp/transformers/auto/tokenizer.py
View File

@@ -57,22 +57,38 @@ else:
("blenderbot", "BlenderbotTokenizer"),
(
"bloom",
("BloomTokenizer", "BloomTokenizerFast" if is_tokenizers_available() else None),
(
"BloomTokenizer",
"BloomTokenizerFast" if is_tokenizers_available() else None,
),
),
("clip", "CLIPTokenizer"),
("codegen", "CodeGenTokenizer"),
("convbert", "ConvBertTokenizer"),
("ctrl", "CTRLTokenizer"),
("distilbert", "DistilBertTokenizer"),
(
"deepseek_v2",
"DeepseekTokenizerFast" if is_tokenizers_available() else None,
),
("electra", "ElectraTokenizer"),
(
"ernie",
("ErnieTokenizer", "ErnieTokenizerFast" if is_tokenizers_available() else None),
(
"ErnieTokenizer",
"ErnieTokenizerFast" if is_tokenizers_available() else None,
),
),
("ernie_m", "ErnieMTokenizer"),
("fnet", "FNetTokenizer"),
("funnel", "FunnelTokenizer"),
("gemma", ("GemmaTokenizer", "GemmaTokenizerFast" if is_tokenizers_available() else None)),
(
"gemma",
(
"GemmaTokenizer",
"GemmaTokenizerFast" if is_tokenizers_available() else None,
),
),
("jamba", "JambaTokenizer"),
("layoutlm", "LayoutLMTokenizer"),
("layoutlmv2", "LayoutLMv2Tokenizer"),
@@ -124,7 +140,10 @@ else:
("unimo", "UNIMOTokenizer"),
(
"gpt",
(("GPTTokenizer", "GPTChineseTokenizer"), "GPTTokenizerFast" if is_tokenizers_available() else None),
(
("GPTTokenizer", "GPTChineseTokenizer"),
"GPTTokenizerFast" if is_tokenizers_available() else None,
),
),
("gau_alpha", "GAUAlphaTokenizer"),
("artist", "ArtistTokenizer"),
@@ -132,7 +151,13 @@ else:
("ernie_vil", "ErnieViLTokenizer"),
("glm", "GLMGPT2Tokenizer"),
("qwen", "QWenTokenizer"),
("qwen2", ("Qwen2Tokenizer", "Qwen2TokenizerFast" if is_tokenizers_available() else None)),
(
"qwen2",
(
"Qwen2Tokenizer",
"Qwen2TokenizerFast" if is_tokenizers_available() else None,
),
),
("yuan", "YuanTokenizer"),
]
)
@@ -158,7 +183,10 @@ def get_configurations():
for class_name, values in TOKENIZER_MAPPING_NAMES.items():
all_tokenizers = get_mapping_tokenizers(values, with_fast=False)
for key in all_tokenizers:
import_class = importlib.import_module(f"paddlenlp.transformers.{class_name}.tokenizer")
try:
import_class = importlib.import_module(f"paddlenlp.transformers.{class_name}.tokenizer")
except ImportError:
import_class = importlib.import_module(f"paddlenlp.transformers.{class_name}.tokenizer_fast")
tokenizer_name = getattr(import_class, key)
name = tuple(tokenizer_name.pretrained_init_configuration.keys())
MAPPING_NAMES[name] = tokenizer_name
@@ -483,7 +511,12 @@ class AutoTokenizer:
"- or the correct path to a directory containing relevant tokenizer files.\n"
)
def register(config_class, slow_tokenizer_class=None, fast_tokenizer_class=None, exist_ok=False):
def register(
config_class,
slow_tokenizer_class=None,
fast_tokenizer_class=None,
exist_ok=False,
):
"""
Register a new tokenizer in this mapping.
@@ -524,4 +557,8 @@ class AutoTokenizer:
if fast_tokenizer_class is None:
fast_tokenizer_class = existing_fast
TOKENIZER_MAPPING.register(config_class, (slow_tokenizer_class, fast_tokenizer_class), exist_ok=exist_ok)
TOKENIZER_MAPPING.register(
config_class,
(slow_tokenizer_class, fast_tokenizer_class),
exist_ok=exist_ok,
)

3
paddlenlp/transformers/blenderbot/tokenizer.py
View File

@@ -16,7 +16,8 @@
from paddle.utils import try_import
from .. import AddedToken, GPTTokenizer
from ..gpt.tokenizer import GPTTokenizer
from ..tokenizer_utils import AddedToken
__all__ = ["BlenderbotTokenizer"]

3
paddlenlp/transformers/codegen/tokenizer.py
View File

@@ -14,7 +14,8 @@
# limitations under the License.
from paddle.utils import try_import
from .. import GPTTokenizer
from ..gpt.tokenizer import GPTTokenizer
__all__ = ["CodeGenTokenizer"]

2
paddlenlp/transformers/conversion_utils.py
View File

@@ -1319,7 +1319,7 @@ class ConversionMixin:
for x in state_keys_real:
if x.endswith(key):
state_keys_map[key] = x
break
# break # remove break for math A.key B.key ...
if key not in state_keys_map:
if not ignore_error:
logger.debug(f"tensor parallel conversion: could not find name {key} in loaded state dict!")

3
paddlenlp/transformers/dallebart/tokenizer.py
View File

@@ -23,7 +23,8 @@ from pathlib import Path
from paddle.utils import try_import
from ...transformers import AddedToken, GPTTokenizer
from ..gpt.tokenizer import GPTTokenizer
from ..tokenizer_utils import AddedToken
__all__ = ["DalleBartTokenizer"]

4
paddlenlp/transformers/deepseek_v2/configuration.py
View File

@@ -42,6 +42,8 @@ class DeepseekV2Config(PretrainedConfig):
Dimension of the MoE representations.
num_hidden_layers (`int`, *optional*, defaults to 32):
Number of hidden layers in the Transformer decoder.
num_nextn_predict_layers (`int`, *optional*, defaults to 1):
Number of nextn predict layers in the DeepSeekV3 Model.
num_attention_heads (`int`, *optional*, defaults to 32):
Number of attention heads for each attention layer in the Transformer decoder.
n_shared_experts (`int`, *optional*, defaults to None):
@@ -135,6 +137,7 @@ class DeepseekV2Config(PretrainedConfig):
intermediate_size=11008,
moe_intermediate_size=1407,
num_hidden_layers=30,
num_nextn_predict_layers=1,
num_attention_heads=32,
num_key_value_heads=32,
n_shared_experts=None,
@@ -180,6 +183,7 @@ class DeepseekV2Config(PretrainedConfig):
self.intermediate_size = intermediate_size
self.moe_intermediate_size = moe_intermediate_size
self.num_hidden_layers = num_hidden_layers
self.num_nextn_predict_layers = num_nextn_predict_layers
self.num_attention_heads = num_attention_heads
self.n_shared_experts = n_shared_experts
self.n_routed_experts = n_routed_experts

177
paddlenlp/transformers/deepseek_v2/modeling.py
View File

@@ -68,6 +68,13 @@ from ..model_outputs import (
from ..model_utils import PretrainedModel, register_base_model
from .configuration import DeepseekV2Config
__all__ = [
"DeepseekV2ForCausalLM",
"DeepseekV2ForSequenceClassification",
"DeepseekV2Model",
"DeepseekV2PretrainedModel",
]
def get_triangle_upper_mask(x, mask=None):
if mask is not None:
@@ -627,6 +634,13 @@ class MoEGate(nn.Layer):
default_initializer=nn.initializer.Constant(1.0),
)
if self.topk_method == "noaux_tc":
self.e_score_correction_bias = paddle.create_parameter(
shape=[self.n_routed_experts],
dtype=paddle.get_default_dtype(),
default_initializer=nn.initializer.Constant(0.0),
)
def forward(self, hidden_states):
bsz, seq_len, h = hidden_states.shape
# compute gating score
@@ -637,17 +651,26 @@ class MoEGate(nn.Layer):
)
if self.scoring_func == "softmax":
with paddle.amp.auto_cast(False):
scores = F.softmax(logits.astype("float32"), axis=-1)
elif self.scoring_func == "sigmoid":
with paddle.amp.auto_cast(False):
scores = F.sigmoid(logits.astype("float32"))
else:
raise NotImplementedError(f"insupportable scoring function for MoE gating: {self.scoring_func}")
# select top-k experts
if self.topk_method == "greedy":
topk_weight, topk_idx = paddle.topk(scores, k=self.top_k, axis=-1, sorted=False)
elif self.topk_method == "group_limited_greedy":
group_scores = scores.reshape([bsz * seq_len, self.n_group, -1]).max(axis=-1).values # [n, n_group]
elif self.topk_method in ["group_limited_greedy", "noaux_tc"]:
if self.topk_method == "group_limited_greedy":
group_scores = scores.reshape([bsz * seq_len, self.n_group, -1]).max(axis=-1).values # [n, n_group]
elif self.topk_method == "noaux_tc":
assert not self.training
scores = scores.reshape([bsz * seq_len, -1]) + self.e_score_correction_bias.unsqueeze(0)
group_scores = (
scores.reshape([bsz * seq_len, self.n_group, -1]).topk(2, axis=-1)[0].sum(axis=-1)
) # [n, n_group]
group_idx = paddle.topk(group_scores, k=self.topk_group, axis=-1, sorted=False)[1] # [n, top_k_group]
group_mask = paddle.zeros_like(group_scores) # [n, n_group]
group_mask.scatter_(1, group_idx, 1) # [n, n_group]
@@ -658,6 +681,7 @@ class MoEGate(nn.Layer):
) # [n, e]
tmp_scores = scores.masked_fill(~score_mask.bool(), 0.0) # [n, e]
topk_weight, topk_idx = paddle.topk(tmp_scores, k=self.top_k, axis=-1, sorted=False)
topk_weight = scores.gather(topk_idx, axis=1) if self.topk_method == "noaux_tc" else topk_weight
# norm gate to sum 1
if self.top_k > 1 and self.norm_topk_prob:
@@ -832,6 +856,9 @@ class DeepseekV2Attention(nn.Layer):
self.o_proj = RowParallelLinear(self.num_heads * self.v_head_dim, self.hidden_size, has_bias=config.attention_bias, input_is_parallel=True)
assert self.num_heads % config.tensor_parallel_degree == 0, f"num_heads: {self.num_heads}, tensor_parallel_degree: {config.tensor_parallel_degree}"
self.num_heads = self.num_heads // config.tensor_parallel_degree
else:
# for without tensor parallel
if self.q_lora_rank is None:
@@ -948,12 +975,6 @@ class DeepseekV2Attention(nn.Layer):
k_nope, value_states = paddle.split(kv, [self.qk_nope_head_dim, self.v_head_dim], axis=-1)
kv_seq_len = value_states.shape[1]
if past_key_value is not None:
if self.layer_idx is None:
raise ValueError(
f"The cache structure has changed since version v4.36. If you are using {self.__class__.__name__} "
"for auto-regressive decoding with k/v caching, please make sure to initialize the attention class "
"with a layer index."
)
kv_seq_len += past_key_value[0].shape[-3]
cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
cos = cos[None, :, None, :]
@@ -1155,7 +1176,7 @@ class DeepseekV2PretrainedModel(PretrainedModel):
]
model_mappings.extend(layer_mappings)
# MoE paramerters
# MoE parameters
model_mappings.append([f"layers.{layer_index}.mlp.gate.weight", None, "transpose"])
for expert_idx in range(config.n_routed_experts):
expert_mappings = [
@@ -1169,11 +1190,10 @@ class DeepseekV2PretrainedModel(PretrainedModel):
model_mappings.append([f"layers.{layer_index}.mlp.shared_experts.down_proj.weight", None, "transpose"])
init_name_mappings(mappings=model_mappings)
# base-model prefix "Qwen2MoEModel"
if "Qwen2Model" not in config.architectures:
if cls.base_model_class.__name__ not in config.architectures:
for mapping in model_mappings:
mapping[0] = "model." + mapping[0]
mapping[1] = "deepseek_v2." + mapping[1]
mapping[1] = f"{cls.base_model_prefix}." + mapping[1]
if not config.tie_word_embeddings:
model_mappings.append(["lm_head.weight", "lm_head.weight", "transpose"])
@@ -1211,12 +1231,15 @@ class DeepseekV2PretrainedModel(PretrainedModel):
# Column Linear
base_actions["layers.0.self_attn.q_proj.weight"] = partial(fn, is_column=True)
base_actions["layers.0.self_attn.q_proj.bias"] = partial(fn, is_column=True)
base_actions["layers.0.self_attn.q_b_proj.weight"] = partial(fn, is_column=True)
# if we have enough num_key_value_heads to split, then split it.
if config.num_key_value_heads % config.tensor_parallel_degree == 0:
base_actions["layers.0.self_attn.k_proj.weight"] = partial(fn, is_column=True)
base_actions["layers.0.self_attn.v_proj.weight"] = partial(fn, is_column=True)
base_actions["layers.0.self_attn.k_proj.bias"] = partial(fn, is_column=True)
base_actions["layers.0.self_attn.v_proj.bias"] = partial(fn, is_column=True)
base_actions["layers.0.self_attn.kv_b_proj.weight"] = partial(fn, is_column=True)
base_actions["layers.0.mlp.up_proj.weight"] = partial(fn, is_column=True)
base_actions["layers.0.mlp.gate_proj.weight"] = partial(fn, is_column=True)
@@ -1261,7 +1284,7 @@ class DeepseekV2PretrainedModel(PretrainedModel):
mean=0.0,
std=self.config.initializer_range
if hasattr(self.config, "initializer_range")
else self.deepseek_v2.config.initializer_range,
else self.config.initializer_range,
shape=layer.weight.shape,
)
)
@@ -1271,7 +1294,7 @@ class DeepseekV2PretrainedModel(PretrainedModel):
mean=0.0,
std=self.config.initializer_range
if hasattr(self.config, "initializer_range")
else self.deepseek_v2.config.initializer_range,
else self.config.initializer_range,
shape=layer.weight.shape,
)
)
@@ -1341,12 +1364,10 @@ class DeepseekV2Model(DeepseekV2PretrainedModel):
# For decoding phase in generation, seq_length = 1, we don't need to add causal mask
if input_shape[-1] > 1:
combined_attention_mask = _make_causal_mask(
input_shape, past_key_values_length=past_key_values_length
input_shape,
past_key_values_length=past_key_values_length,
)
if get_env_device() == "npu":
expanded_attn_mask = expanded_attn_mask.astype("bool") & combined_attention_mask.astype("bool")
else:
expanded_attn_mask = expanded_attn_mask & combined_attention_mask
expanded_attn_mask = expanded_attn_mask & combined_attention_mask
# [bsz, seq_len, seq_len] -> [bsz, 1, seq_len, seq_len]
elif len(attention_mask.shape) == 3:
expanded_attn_mask = attention_mask.unsqueeze(1).astype("bool")
@@ -1354,20 +1375,19 @@ class DeepseekV2Model(DeepseekV2PretrainedModel):
else:
expanded_attn_mask = attention_mask
else:
expanded_attn_mask = _make_causal_mask(input_shape, past_key_values_length=past_key_values_length)
expanded_attn_mask = _make_causal_mask(
input_shape,
past_key_values_length=past_key_values_length,
)
# Convert bool attention_mask to float attention mask, which will be added to attention_scores later
if get_env_device() == "npu":
if get_env_device() == "xpu":
x = paddle.to_tensor(0.0, dtype="float32")
y = paddle.to_tensor(paddle.finfo(dtype).min, dtype="float32")
expanded_attn_mask = expanded_attn_mask.astype("float32")
expanded_attn_mask = paddle.where(expanded_attn_mask, x, y).astype(dtype)
elif get_env_device() in ["xpu", "gcu"]:
x = paddle.to_tensor(0.0, dtype=dtype)
y = paddle.to_tensor(paddle.finfo(dtype).min, dtype=dtype)
expanded_attn_mask = expanded_attn_mask.astype(dtype)
expanded_attn_mask = paddle.where(expanded_attn_mask, x, y).astype(dtype)
y = paddle.to_tensor(-1.7005809656952787e38, dtype="float32")
expanded_attn_mask = paddle.where(expanded_attn_mask, x, y)
else:
expanded_attn_mask = paddle.where(expanded_attn_mask, 0.0, paddle.finfo(dtype).min).astype(dtype)
expanded_attn_mask = paddle.where(expanded_attn_mask.cast("bool"), 0.0, paddle.finfo(dtype).min).astype(
dtype
)
return expanded_attn_mask
@paddle.jit.not_to_static
@@ -1611,9 +1631,7 @@ class DeepSeekV2LMHead(nn.Layer):
if tensor_parallel_output is None:
tensor_parallel_output = self.config.tensor_parallel_output
logits = parallel_matmul(
hidden_states, self.weight, transpose_y=False, tensor_parallel_output=tensor_parallel_output
)
logits = parallel_matmul(hidden_states, self.weight, tensor_parallel_output=tensor_parallel_output)
return logits
@@ -1622,9 +1640,10 @@ class DeepseekV2ForCausalLM(DeepseekV2PretrainedModel):
def __init__(self, config: DeepseekV2Config):
super().__init__(config)
self.config = config
self.deepseek_v2 = DeepseekV2Model(config)
self.vocab_size = config.vocab_size
self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias_attr=False)
self.lm_head = DeepSeekV2LMHead(config)
self.criterion = DeepSeekV2PretrainingCriterion(config)
def get_input_embeddings(self):
@@ -1723,49 +1742,14 @@ class DeepseekV2ForCausalLM(DeepseekV2PretrainedModel):
)
def prepare_inputs_for_generation(
self,
input_ids,
past_key_values=None,
attention_mask=None,
inputs_embeds=None,
**kwargs,
self, input_ids, use_cache=False, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs
):
if past_key_values is not None:
if isinstance(past_key_values, Tuple[paddle.Tensor]):
cache_length = past_key_values.get_seq_length()
past_length = past_key_values.seen_tokens
max_cache_length = past_key_values.get_max_length()
else:
cache_length = past_length = past_key_values[0][0].shape[2]
max_cache_length = None
# Keep only the unprocessed tokens:
# 1 - If the length of the attention_mask exceeds the length of input_ids, then we are in a setting where
# some of the inputs are exclusivelly passed as part of the cache (e.g. when passing input_embeds as
# input)
if attention_mask is not None and attention_mask.shape[1] > input_ids.shape[1]:
input_ids = input_ids[:, -(attention_mask.shape[1] - past_length) :]
# 2 - If the past_length is smaller than input_ids', then input_ids holds all input tokens. We can discard
# input_ids based on the past_length.
elif past_length < input_ids.shape[1]:
input_ids = input_ids[:, past_length:]
# 3 - Otherwise (past_length >= input_ids.shape[1]), let's assume input_ids only has unprocessed tokens.
# If we are about to go beyond the maximum cache length, we need to crop the input attention mask.
if (
max_cache_length is not None
and attention_mask is not None
and cache_length + input_ids.shape[1] > max_cache_length
):
attention_mask = attention_mask[:, -max_cache_length:]
position_ids = kwargs.get("position_ids", None)
if attention_mask is not None and position_ids is None:
# create position_ids on the fly for batch generation
position_ids = attention_mask.long().cumsum(-1) - 1
position_ids.masked_fill_(attention_mask == 0, 1)
if past_key_values:
position_ids = position_ids[:, -input_ids.shape[1] :]
batch_size, seq_length = input_ids.shape
position_ids = kwargs.get("position_ids", paddle.arange(seq_length).expand((batch_size, seq_length)))
attention_mask = kwargs.get("attention_mask", None)
if past_key_values:
input_ids = input_ids[:, -1].unsqueeze(axis=-1)
position_ids = position_ids[:, -1].unsqueeze(-1)
# if `inputs_embeds` are passed, we only want to use them in the 1st generation step
if inputs_embeds is not None and past_key_values is None:
@@ -1777,12 +1761,49 @@ class DeepseekV2ForCausalLM(DeepseekV2PretrainedModel):
{
"position_ids": position_ids,
"past_key_values": past_key_values,
"use_cache": kwargs.get("use_cache"),
"use_cache": use_cache,
"attention_mask": attention_mask,
}
)
return model_inputs
def _get_model_inputs_spec(self, dtype: str):
return {
"input_ids": paddle.static.InputSpec(shape=[None, None], dtype="int64"),
"attention_mask": paddle.static.InputSpec(shape=[None, None], dtype="int64"),
"position_ids": paddle.static.InputSpec(shape=[None, None], dtype="int64"),
}
@staticmethod
def update_model_kwargs_for_generation(outputs, model_kwargs, is_encoder_decoder=False):
# update cache
if isinstance(outputs, tuple) and len(outputs) > 1 and not isinstance(outputs[1], paddle.Tensor):
model_kwargs["past_key_values"] = outputs[1]
if isinstance(outputs, CausalLMOutputWithPast) and "past_key_values" in outputs:
model_kwargs["past_key_values"] = outputs.past_key_values
# update position_ids
if "position_ids" in model_kwargs and model_kwargs["position_ids"] is not None:
position_ids = model_kwargs["position_ids"]
model_kwargs["position_ids"] = paddle.concat([position_ids, position_ids[..., -1:] + 1], axis=-1)
if not is_encoder_decoder and "attention_mask" in model_kwargs:
# TODO: support attention mask for other models
attention_mask = model_kwargs["attention_mask"]
if len(attention_mask.shape) == 2:
model_kwargs["attention_mask"] = paddle.concat(
[attention_mask, paddle.ones([attention_mask.shape[0], 1], dtype=attention_mask.dtype)],
axis=-1,
)
elif len(attention_mask.shape) == 4:
model_kwargs["attention_mask"] = paddle.concat(
[attention_mask, paddle.ones([*attention_mask.shape[:3], 1], dtype=attention_mask.dtype)],
axis=-1,
)[:, :, -1:, :]
return model_kwargs
@staticmethod
def _reorder_cache(past_key_values, beam_idx):
reordered_past = ()

4
paddlenlp/transformers/deepseek_v2/tokenizer_fast.py
View File

@@ -15,6 +15,10 @@ from typing import List, Optional, Union
from ..llama import LlamaTokenizerFast
__all__ = [
"DeepseekTokenizerFast",
]
class DeepseekTokenizerFast(LlamaTokenizerFast):
def convert_ids_to_tokens(

16
paddlenlp/transformers/deepseek_v3/__init__.py Normal file
View File

@@ -0,0 +1,16 @@
# Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from .configuration import *
from .modeling import *

33
paddlenlp/transformers/deepseek_v3/configuration.py Normal file
View File

@@ -0,0 +1,33 @@
# Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved.
# Copyright 2023 Mistral AI and the HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" DeepSeekV3 model configuration"""
from ..deepseek_v2.configuration import DeepseekV2Config
__all__ = [
"DeepseekV3Config",
]
class DeepseekV3Config(DeepseekV2Config):
model_type = "deepseek_v3"
keys_to_ignore_at_inference = ["past_key_values"]
def __init__(
self,
**kwargs,
):
super().__init__(
**kwargs,
)

167
paddlenlp/transformers/deepseek_v3/modeling.py Normal file
View File

@@ -0,0 +1,167 @@
# Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved.
# Copyright 2023 DeepSeek-AI and The HuggingFace Inc. team. All rights reserved.
#
# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
# and OPT implementations in this library. It has been modified from its
# original forms to accommodate minor architectural differences compared
# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Paddle DeepSeek model."""
from __future__ import annotations
from typing import List, Optional, Tuple, Union
import paddle
from ..deepseek_v2.modeling import (
DeepseekV2ForSequenceClassification,
DeepSeekV2LMHead,
DeepseekV2Model,
DeepseekV2PretrainedModel,
DeepSeekV2PretrainingCriterion,
)
from ..model_outputs import CausalLMOutputWithPast
from ..model_utils import register_base_model
from .configuration import DeepseekV2Config
__all__ = [
"DeepseekV3ForCausalLM",
"DeepseekV3ForSequenceClassification",
"DeepseekV3Model",
"DeepseekV3PretrainedModel",
]
class DeepseekV3PretrainedModel(DeepseekV2PretrainedModel):
config_class = DeepseekV2Config
base_model_prefix = "deepseek_v3"
_no_split_modules = ["DeepseekV2DecoderLayer"]
@register_base_model
class DeepseekV3Model(DeepseekV2Model):
def __init__(self, config: DeepseekV2Config):
super().__init__(config)
class DeepseekV3ForCausalLM(DeepseekV3PretrainedModel):
_tied_weights_keys = ["lm_head.weight"]
def __init__(self, config: DeepseekV2Config):
super().__init__(config)
self.deepseek_v3 = DeepseekV3Model(config)
self.vocab_size = config.vocab_size
self.lm_head = DeepSeekV2LMHead(config)
self.criterion = DeepSeekV2PretrainingCriterion(config)
def get_input_embeddings(self):
return self.deepseek_v3.embed_tokens
def set_input_embeddings(self, value):
self.deepseek_v3.embed_tokens = value
def get_output_embeddings(self):
return self.lm_head
def set_output_embeddings(self, new_embeddings):
self.lm_head = new_embeddings
def set_decoder(self, decoder):
self.deepseek_v3 = decoder
def get_decoder(self):
return self.deepseek_v3
def forward(
self,
input_ids: paddle.Tensor = None,
attention_mask: Optional[paddle.Tensor] = None,
position_ids: Optional[paddle.Tensor] = None,
past_key_values: Optional[List[paddle.Tensor]] = None,
inputs_embeds: Optional[paddle.Tensor] = None,
labels: Optional[paddle.Tensor] = None,
use_cache: Optional[bool] = None,
output_attentions: Optional[bool] = None,
output_hidden_states: Optional[bool] = None,
return_dict: Optional[bool] = None,
) -> Union[Tuple, CausalLMOutputWithPast]:
r"""
Args:
labels (`paddle.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
Labels for computing the masked language modeling loss. Indices should either be in `[0, transformers.,
config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
(masked), the loss is only computed for the tokens with labels in `[0, transformers., config.vocab_size]`.
Returns:
Example:
```python
>>> from transformers import AutoTokenizer, DeepseekV3ForCausalLM
>>> model = DeepseekV3ForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS)
>>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER)
>>> prompt = "Hey, are you conscious? Can you talk to me?"
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> # Generate
>>> generate_ids = model.generate(inputs.input_ids, max_length=30)
>>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
"Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
```"""
output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
output_hidden_states = (
output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
)
return_dict = return_dict if return_dict is not None else self.config.use_return_dict
# decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
outputs = self.deepseek_v3(
input_ids=input_ids,
attention_mask=attention_mask,
position_ids=position_ids,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
use_cache=use_cache,
output_attentions=output_attentions,
output_hidden_states=output_hidden_states,
return_dict=return_dict,
)
hidden_states = outputs[0]
logits = self.lm_head(hidden_states)
loss = None
# TODO@DrownFish19: shift labels
if labels is not None:
loss = self.criterion(logits, labels)
if not return_dict:
output = (logits,) + outputs[1:]
return (loss,) + output if loss is not None else output
return CausalLMOutputWithPast(
loss=loss,
logits=logits,
past_key_values=outputs.past_key_values,
hidden_states=outputs.hidden_states,
attentions=outputs.attentions,
)
class DeepseekV3ForSequenceClassification(DeepseekV2ForSequenceClassification):
def __init__(self, config):
super().__init__(config)

15
paddlenlp/transformers/ernie_gen/modeling.py
View File

@@ -21,16 +21,13 @@ import six
from paddle import nn
from paddle.nn import functional as F
from paddlenlp.transformers import (
BertPretrainedModel,
ElectraPretrainedModel,
ErniePretrainedModel,
RobertaPretrainedModel,
)
from paddlenlp.utils.download import resolve_file_path
from paddlenlp.utils.log import logger
from ...utils.download import resolve_file_path
from ...utils.log import logger
from .. import PretrainedModel, register_base_model
from ..bert.modeling import BertPretrainedModel
from ..electra.modeling import ElectraPretrainedModel
from ..ernie.modeling import ErniePretrainedModel
from ..roberta.modeling import RobertaPretrainedModel
from ..utils import InitTrackerMeta, fn_args_to_dict
__all__ = ["ErnieGenPretrainedModel", "ErnieForGeneration", "ErnieGenModel"]

3
paddlenlp/transformers/glm/tokenizer.py
View File

@@ -23,7 +23,8 @@ import sentencepiece as spm
from scipy.linalg import block_diag
from ...utils.log import logger
from .. import BertTokenizer, GPTTokenizer
from ..bert.tokenizer import BertTokenizer
from ..gpt.tokenizer import GPTTokenizer
from ..tokenizer_utils import PretrainedTokenizer
from ..tokenizer_utils_base import BatchEncoding

2
paddlenlp/transformers/luke/tokenizer.py
View File

@@ -27,7 +27,7 @@ import sys
import warnings
from itertools import repeat
from .. import RobertaBPETokenizer
from ..roberta.tokenizer import RobertaBPETokenizer
try:
from functools import lru_cache

11
paddlenlp/transformers/roberta/tokenizer.py
View File

@@ -21,14 +21,9 @@ from paddle.utils import try_import
from paddlenlp.utils.download import resolve_file_path
from .. import (
AddedToken,
BasicTokenizer,
GPTTokenizer,
PretrainedTokenizer,
WordpieceTokenizer,
)
from ..gpt.tokenizer import bytes_to_unicode
from ..bert.tokenizer import BasicTokenizer, WordpieceTokenizer
from ..gpt.tokenizer import GPTTokenizer, bytes_to_unicode
from ..tokenizer_utils import AddedToken, PretrainedTokenizer
__all__ = ["RobertaTokenizer", "RobertaChineseTokenizer", "RobertaBPETokenizer"]